Next major for the ecosystem

[wooorm]

Soon (April), it’s time to switch to ESM (without CJS backup) for all packages in the ecosystem.
That’s a lot of projects switching over, and a round of majors that bubbles from the bottom through to the top.

I’ve had some experience with that recently in a push in micromark (with CJS fallback, causing issues). My recent own projects dioscuri and xdm are fully ESM. It’s going to be a lot of fun for the ecosystem 😅

Otherwise, here are some changes slate for this next bubbling:

Big breaking changes:

• archive and deprecate remark plugins that (should) have rehype equivalents, and add some docs somewhere on whether folks should use remark or rehype plugins (currently quite confusing)
  • remarkjs/remark-external-links
  • remarkjs/remark-unwrap-images
  • remarkjs/remark-highlight.js
  • remarkjs/remark-react
  • remarkjs/remark-autolink-headings
  • remark-html-katex in remarkjs/remark-math
• archive and deprecate projects w/ very few downloads:
  • remarkjs/remark-midas
  • remarkjs/remark-defsplit
  • retextjs/retext-sentiment
  • vfile/vfile-reporter-folder-json
  • remarkjs/grunt-remark
  • syntax-tree/unist-builder-blueprint
  • syntax-tree/unist-builder-blueprint-cli
  • syntax-tree/hast-util-to-snabbdom
  • syntax-tree/mdast-util-from-quill-delta
• remark-yaml-config: update js-yaml
• vfile-matter: update js-yaml
• unified-engine: update js-yaml
• vfile-message: rename location -> position (Rename location field to position vfile/vfile-message#6)
• vfile: rename contents to value (Rename contents field to value vfile/vfile#49)
• mdast-util-to-string: do not use node.title (Remove node.title support syntax-tree/mdast-util-to-string#8)
• mdast: rename depth to rank (Rename depth to rank syntax-tree/mdast#31) (I feel the other issues there plus Rename node types syntax-tree/nlcst#6 are too much break for what they bring)
• hast: change doctypes (Simplify doctypes syntax-tree/hast#19)

Small breaking changes:

• hastscript’s h and s and a JSX pragma Add support for using h, s as a JSX pragmas syntax-tree/hastscript#15
• unist-util-find-all-before’s results should be reversed probably Results are reversed syntax-tree/unist-util-find-all-before#6

For ES++ features (unifiedjs/rfcs#4), I’d say we can do that after we switch to ESM, because ESM will already be a lot of work, and it gives us a baseline of what ES features / engines to support.

[chrisrzhou]

Are there discussion threads (any links to notable discussions are good enough!) where I can read more into the motivation to move formally to using ESM? Inquiring to stay up to date with the unified ecosystem, and also getting context in the broader JS ecosystem.

[wooorm]

micromark/micromark#27 and other PRs there, but that was both ESM and CJS.
See also https://twitter.com/sindresorhus/status/1349294527350149121.
The motivation is essentially: we’ll have to move some time. This is a good time!

[nnmrts]

Great to see unified switching over as well!

@wooorm Please don't archive retextjs/retext-sentiment. I'm ready to maintain it if you want. It just seems too useful to me to just forget it.

[wooorm]

I’m also going to try to add jsdoc based types to everything (starting with ±150 of my own projects).

Another big one is to add support for plugins as ESM in unified-engine: the only way forward I see is if plugins export default? Or have a named plugin export? 🤔 I prefer the first from the unified-engine perspective, but as an author I don’t like default exports.

[chrisrzhou]

@wooorm, (for my own learning), a few questions relating to his upcoming refactor:

1. What are the benefits for jsdoc types vs .d.ts vs strictly implementing the code in TS?

The main benefits I'm inferring is decoupling the dependency and reliance on TS, so that if the ecosystem needs to be free of TS in the future, we at least have JSDocs with an easy path to decoupling in the future? I'm curious about the decision because I'm facing similar concerns with trying to keep source code as independent from tooling/dependencies as possible (flow was king at one point, typescript is currently 'king', but no one knows what will happen with the typing ecosystem in the future).

2. With the focus on ES modules, it seems pretty amazing that we don't need build tools since it's natively supported by Node and modern browsers. How do you resolve dependencies in unified ecosystem that are not migrated to ES modules? E.g. we cannot naturally rely on package.json's type: module if an upstream dependency hasn't formally migrated to ES module (even if the package supports ES modules through build tools and exporting "module: index.module.js" in package.json, Node scripts won't treat it as a formal ES module). Is this problem avoided in the ecosystem because the ecosystem has very limited external dependencies?

3. Related to 2, should packages (even microutils) in the ecosystem care about bundling/minifying or should that just be left to upstream consumers?

Would love to hear the perspectives and approaches of the team here, so I can also learn something from it!

And my (personal and likely not-important) opinion on:

Another big one is to add support for plugins as ESM in unified-engine: the only way forward I see is if plugins export default? Or have a named plugin export? 🤔 I prefer the first from the unified-engine perspective, but as an author I don’t like default exports.

I think most authors avoid default exports and use named exports (explicit named export is useful as a static guard, also great for refactoring), although I think that if we have to consider how consumers would use an API, we should make the appropriate decision based on how this library is intended to be consumed. For myself, I use 1) named exports for internal implementation all the time and 2) decide the final public export based on what makes sense for the consumer.

[chrisrzhou]

Didn't realize Github does not support comment threads, so it's kind of annoying keeping track of discussions and responses :P (this comment further adds to the problem)

[wooorm]

What are the benefits for jsdoc types vs .d.ts vs strictly implementing the code in TS?

I strongly dislike having to compile things. I think the benefit of JS is that it just runs everywhere. If you’re going to compile, IMO there are much nicer languages. I don’t write these projects in JS because I like JS per se (although I do), I do it because it works everywhere.
And, these projects are all long lived: I’ve been maintaining most stuff for 7+ years already. Rewriting the whole ecosystem for every new hype is a lot of work.

This is also why is dislike “modern” javascript. One such example of this is how the projects that’ve pushed ESM over the years, actually don’t support ESM. Webpack, snowpack configs, Jest: they all don’t get actual ESM.
I’m not adverse to types per se, and especially would like to use their information for docs. But it’s rather complex to try to make a single, low-config, doc system based on them that works nicely for all my projects, so maybe that’s for the next iteration?

We (@ChristianMurphy and I) did run into a couple places where JSDoc based TS doesn’t cut it yet, though.
And while some of it might be me being a TS beginner, I do have some doubts. It’s adding a lot of complexity for my style of small, well-tested projects, but not adding a lot of benefits.

So, you’re seeing some of the same things I’m seeing, but I hope the above explains a bit more about my current state of reasoning

With the focus on ES modules, it seems pretty amazing that we don't need build tools since it's natively supported by Node and modern browsers.

Omgosh! It is! Of course, bundling would be needed in most professional settings. But it works. And it works nicely.

How do you resolve dependencies in unified ecosystem that are not migrated to ES modules

Using ESM from CJS is a bit hard, you have to use an import(x) expression.
The other way around is smooth sailing: you can import CJS and ESM mostly seamlessly from ESM.

Node scripts won't treat it as a formal ES module

You can set "type": "module" in a package.json and everything under it will be seen as ESM, and CJS otherwise. Or you can opt-in/out by using .mjs or .cjs extensions.

should packages (even microutils) in the ecosystem care about bundling/minifying or should that just be left to upstream consumers?

Can you expand on this? IMO, all projects that are “universal” (node + browser) should care about size.
But if you’re asking about dual-CJS/ESM, IMO it’s too much work and too buggy to try it. Pick ESM.

For myself, I use 1) named exports for internal implementation all the time and 2) decide the final public export based on what makes sense for the consumer.

I agree on not using default exports for libraries: while names are a bit superfluous in most of my projects (as they export one thing), default exports have downsides.
But plugins have to be importable by tools too. So it has to be either a single named identifier (plugin?) or the default export. Using plugin for every plugin is just as well a bad experience for users that do import it manually. So that’s not my preference.
Babel and Rollup do support ESM plugins nicely, and they use default exports, so there’s some precedence for it too!

[chrisrzhou]

Thanks for the detailed response, it helps affirm some of the dilemmas I've been having around 'modern' JS.

I 100% agree with a lot of the statements you (for rest of the post, "you" refers to the unified team! ) made about compiling, typing, and in general coupling standard JS source code with build tools. My experience with JS has largely been a follower and consumer of such tools, and only recently by following and studying code in various repos (e.g. unified, three, tape), did I start paying attention why authors take on the cost of maintaining 'buildless' and 'typeless' libraries.

On typing, TS does offer a much delightful developer experience, but I think the danger is the community being over-eager in thinking that TS guarantees type-safe code. TS doesn't protect against runtime invalid types, and in the end it just boils down to a simple question if consumers are using the APIs as-intended. I personally think TS improves the developer experience and exploration of code, but I think that is as far the value I can see with it, and I agree with your principles on avoiding rewriting libraries if hyped-package@will.eventually.be.outclassed actually gets in the way of authoring source code. I think the JSDoc with TS typings is a safe decoupled approach that you can toss out TS and still keep readable documentation (I'm assuming unified's JSdoc strings are sourcing from TS types?).

On build tools, babel and webpack etc are powerful and have allowed developers to write basically any syntax and rely on transpiling to arrive to functioning JS. This has allowed developers to be expressive, and caused a huge momentum and explosion in the JS ecosystem, but I also tend to see this as a deviation from specs/standards-driven development. In the end, the real standards are those of the language and ecosystem (browsers). Everything else is a standard on top of another standard, and none of these will last. We've moved from grunt- > gulp -> webpack -> rollup -> snowpack -> something else but we're really just playing on top of the base standard of writing code that 'just works'. Authoring code with new syntax sugar is sweet (pun intended), but just as coffeescript eventually fell out, so would any source code authored in potentially-non-compliant ECMA proposals fall out too. And this all deteriorates to teams having no choice but to author code in an outdated/incorrect standard, or basically having to push the "refactor everything" red button at some point. The cost of delightful development is eventually paid with a rewrite, something that you've alluded to.

Anyway, the above is just myself chatting and responding to something that I've slowly come expose to and appreciate because of the discussions and journey in learning unified and its implementation. Back to the specific questions that I think I may not have expressed very well, so structuring it a little better:

Question 1:
If I were writing ESM packages intended for ESM consumers, should I care about exporting a .cjs export, and also care about minified exports or should I leave it to consumers to decide how to handle this? Will unified ecosystem export .cjs bundles, minified bundles etc, which would basically means that there is a compile/build process for non-ESM outputs? Or is unified ecosystem going to pure and completely compile-/build-free which means limiting the ecosystem to ESM-only consumers?

Question 2:
(see below code snippet for detail) If my package depends on an NPM dependency that isn't written in standard ESM, but does provide an ESM output, this would not work with node ./my-esm-script.js. Does unified packages face this problem or it is irrelevant because the ecosystem is self-contained (no external dependencies)? If this problem exists for unified, how can this be solved?

my-esm-script.js

import x from 'vendor-library';

x();

vendor-library/package.json

 {
  "module": "dist/index.module.js",
  "main": "dist/index.js",
 }

Note that there is NO "type": "module" field because the author of vendor-library did not set it up that way since it wasn't a library developed as a pure ESM module, but rather the final dist files are generated through some build tool (e.g. microbundle, rollup etc).

Running node ./my-esm-script.js will throw an error because

• node will do what node does: check for "type": "module" or .mjs extension to assume that a module is ESM.
• In this case, vendor-library/package.json is not a valid ESM package for two reasons:
  • "type: module" isn't specified
  • "main: dist/index.js" isn't with .mjs extension (expected because it's meant to be a CJS export by convention)
• Neither can node know to pick up that module: "dist/index.module.js" should be used because there's nothing to indicate that this is an ESM, so it instead picks up "main": "dist/index.js", which as you would have guessed, creates conflicts when my-esm-script is ES-based but dist/index.js is CJS-based.

In the above example, I cannot control setting "type": "module" in vendor-library/package.json because the vendor owns the implementation. I believe that unless the vendor library follows the ESM standards, there is no way I can consume this vendor package in a pure ESM way (without build/compile steps)?

[nnmrts]

In the above example, I cannot control setting "type": "module" in vendor-library/package.json because the vendor owns the implementation. I believe that unless the vendor library follows the ESM standards, there is no way I can consume this vendor package in a pure ESM way (without build/compile steps)?

I went on a bit of a tangent here, based on a misunderstanding of your second question, @chrisrzhou. Scroll down to see my attempt to answer that and the first one.

I feel you and that's the one big thing I'm always unsure on how to solve it. I really like how the ecosystem and community in general is finally but slowly moving to the methodology you described, the somewhat naive philosophy to at least try to write "future-proof" code. To follow the standard, to do things that work everywhere. Well at least everywhere a developer expects it to work.

But, yes, you are totally right, at some point, you also just have to accept that projects progress with wildly different velocities and manners to that desired pure ESM utopia. Some will even never get there, because they are either unmaintained, or the respective maintainer doesn't believe in that already mentioned philosophy. People have different backgrounds and expectations of how code should be interacted with, what software should do and what it shouldn't do, and that's fine. From a personal perspective, for example, I would never use Typescript in the context of coding modules and projects that are meant to be used by other fellow developers, and I don't support or share the philosophy behind the language at all. People can shout at me all they want about its "benefits", I know about them, I worked with Typescript professionally, but as long as it isn't (and shouldn't be) directly interpreted by browsers and runtimes and it still needs to be compiled, as long as it just follows a standard that's ultimately controlled by a single company instead of at least a couple of companies, important developers we kind of trust and a transparent well-thought-out process of introducing new features, it just makes no sense to me why I should use it in a community context.

But without going to much into depth about all that, I don't think the majority of unifieds modules depend or have to depend on such CJS vendor-libraries you mentioned. I didn't go through all of the hundreds of package.jsons, but as far as I know, most of them are themselves that vendor-library you will eventually depend on, the fundamental building blocks for bigger projects.

---

Question 1:

Most of unifieds projects, correct me if I'm wrong, also aren't meant to be used directly in a browser, they are meant to be used by other developers and their projects. And I understand the accessibility concerncs, the aspiration of deploying your project in all imaginable formats to please the most amount of developers possible. But the answer to accessibility ends where the question of complacency starts. I think @wooorm ultimately decides which way unified will go, but I definitely think it, and - trying to answer your first question - anyone writing "ESM packages intended for ESM consumers", should "leave it to consumers to decide how to handle this".

Question 2:

Your second question has multiple layers though. If we are talking about Node.js and the general context of unified in its current state, no, there aren't any conflicts.

Running node ./my-esm-script.js will not throw an error, importing CJS in ESM (in Node.js) is totally fine. See this super simple example: https://github.com/nnmrts/esm-test which imports https://github.com/nnmrts/cjs-test. But yes, this won't import the ESM format file, the index.module.js. It will use the usual, main CJS index.js file. If that is what you are referring to, true, you won't get the whole ESM "experience", but does that really matter? Your own code is still in ESM. So no, unified doesn't and never will face a "problem" here, as long as it sticks to Node.js.

If you are referring to the more general context of "real" ESM, independent of Node.js, then yes, there would be an error if you import the cjs file. Since ESM also doesn't allow you to just do import x from "vendor-library"; (you have to refer to an actual file), you would ideally refer to the ESM format file from that vendor-library directly, like:

import x from "vendor-library/dist/index.module.js";

Or actually, in this case:

import x from "./node_modules/vendor-library/dist/index.module.js";

Do I personally believe unified (and similar projects for what it's worth) should be Node.js-independent and use the "real" ESM format of importing actual files, maybe even by using Deno and publishing to nest.land? Yes absolutely. Do I think it would be healthy to do this at the same time it switches to ESM syntax and output in general? Nope. I would suggest to focus on that first, to make migration easier and especially since "pure" ESM, Deno, nest.land and all that stuff is still pretty new territory for most people.

[wooorm]

@chrisrzhou

If I were writing ESM packages intended for ESM consumers, should I care about exporting a .cjs export, and also care about minified exports or should I leave it to consumers to decide how to handle this? Will unified ecosystem export .cjs bundles, minified bundles etc, which would basically means that there is a compile/build process for non-ESM outputs? Or is unified ecosystem going to pure and completely compile-/build-free which means limiting the ecosystem to ESM-only consumers?

I believe source code should be readable. I don’t see a lot of value in publishing minified bundles.
Dual packages, so both ESM and CJS, are again complex, but importantly also come with quite a hazard.

There are cases where, based on the ecosystem (browser, node, react-native, etc), or based on other environmental things (dev vs. production), different code should run. These “conditions” are being added to Node. In my RSC demo I found that the React team was experimenting with them for server vs. client.
These conditions are probably exceptions though.

If my package depends on an NPM dependency that isn't written in standard ESM, but does provide an ESM output, this would not work with node ./my-esm-script.js. Does unified packages face this problem or it is irrelevant because the ecosystem is self-contained (no external dependencies)? If this problem exists for unified, how can this be solved?

The module field shown here is not “standardized” by Node, indeed. It’s an instruction for faux-ESM bundlers such as webpack.

I suggest for folks that maintain semi-inactive projects/apps to stick with what they have and not update dependencies moving to ESM.
After a couple of warts in Jest/Electron/webpack/Next/CRA are solved, hopefully in a few months, I think it’s fine for folks of semi-active or active projects to move to ESM: they can update their dependencies, and use CJS packages just fine.

So, for “Does unified packages face this problem”: No, unified does not face such a problem, because switching over to ESM solves everything.

---

@nnmrts

I think @wooorm ultimately decides which way unified will go

I do decide a lot through sheer activity and also some level of seniority in this ecosystem, but it’s a democratic process: https://github.com/unifiedjs/collective/blob/main/decisions.md

---

And interesting for you both: in the future we’ll get import maps in browsers: https://github.com/WICG/import-maps. That allows these “bare” specifiers (package, package/file.js, @scope/page) to work there too.
I’m imagining that a tool will be made to generate such an export map from node_modules/, so that at dev-time it work just as seamlessly in a browser as on Node, and then everything is bundled together for production.

EDIT: And, the inverse is also being worked on (but experimental) in Node, with Node loaders. Here’s how to import http:// and such in Node: https://github.com/node-loader/node-loader-http

[chrisrzhou]

Seems like a really disruptive/hectic period for JS developers, but hoping the equilibrium settles to a simpler and standard ESM with less build tools. Would be great if more JS code is just standard JS code! Thanks for all the detailed response and info!

[aminya]

For the packages that have other dependencies, and there is no transpilation happening, we cannot use Pure ESM packages because:

• A huge portion of the packages will not migrate.
• Node doesn't support seamlessly mixing require and import:
  • It is not possible to require ESM modules in a CJS package
  • There is no alternative for dynamic synchronous CJS requires
  • If the functions are all synchronous (e.g. constructors must be sync), the above leads to importing everything in advance at the top of the module, which hurts the performance.

[wooorm]

The above is incorrect because:

• From ESM, you can import CJS without any problems: import whatever from 'some-commonjs-package' works
• From CJS, you can import ESM async: whatever = (await import('some-esm-package')).identifier

[benrbray]

I want to chime in here and say that while I really love using Remark and appreciate the work put into it, getting the ESM modules to work TS is an absolute pain. I get wanting to move forward with an ES2015 feature that has had over six years now to catch on, but the simple fact is that the rest of the ecosystem still isn't ready. Until then, I really wish that Unified would change its mind about the decision not to include backward-compatible packages.

I'm using remark in an app built with TS + webpack + electron. With this combination of tools, some of the unified packages (mostly, for tree manipulation) simply don't work. Instead, I ended up either copying the unified source files manually into my own tree and rewriting the imports, or rewriting the functionality myself in pure TS (the unified API currently likes to wrap everything up in all-but-the-kitchen-sink functions, which are difficult to write types for, as evidenced by the current JSDoc strings).

On a positive note, I'm glad that unified is flexible enough that I can write my own solutions when needed. I just wish I didn't have to.

[wooorm]

Folks, it’s fine to talk about ESM. Either here or in other places. But this issue is about moving to ESM and other things in majors. And the discussion on whether to do it or not was had in Feb and March.
Christian and I are trying to keep the thread manageable by hiding stuff that is irrelevant for the majors while still answering questions and pointing towards better places to discuss ESM.

@benrbray I don’t think dual ESM/CJS is viable for the JavaScript ecosystem: https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#gistcomment-3851022.

Re TS: one part of the majors is also that everything is (strongly) typed, externally but also inside projects themselves. So that at least should help you a lot when you can move to ESM in TS.

[ChristianMurphy]

I'd second @wooorm's comment in #121 (comment) and would add:

I'm using remark in an app built with TS + webpack + electron. With this combination of tools, some of the unified packages (mostly, for tree manipulation) simply don't work

It may be worth opening a discussion to talk through this (https://github.com/unifiedjs/.github/blob/main/support.md#questions).
With https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#im-having-problems-with-esm-and-typescript and https://gist.github.com/sindresorhus/a39789f98801d908bbc7ff3ecc99d99c#how-can-i-import-esm-in-electron it should be possible to run the latest version on Electron.

I ended up either copying the unified source files manually into my own tree and rewriting the imports, or rewriting the functionality myself in pure TS

In the latest version all utilities are pure TypeScript.
TypeScript offers two syntaxes, annotation syntax and JSDoc syntax, unified primarily uses the latter. (unifiedjs/rfcs#5)

[wooorm]

Done! Alright, folks, everything is now ESM and typed! The packages up for deprecation were deprecated. I’ve “soft deprecated” certain packages by first updating them and then marking them as legacy, like so, to prevent too much churn. They can be fully deprecated later but this makes it a bit easier to still land security fixes when needed.
I’ve also refrained from a couple of other changes such as depth -> rank for mdast headings.

If you have questions about ESM, you might find answers in this Gist.
Generally, the problems will be with build tools, bundlers, and site generators, so you might also raise questions or find answers in their corresponding issue trackers or discussion channels.
If your question relates to unified packages, feel free to open a discussion with a reproduction in one of our boards.