Future: ESM version, new streams, more runtimes

[uhop]

I am opening it as a tracker in place of #97.

And it is a place to discuss the topics of ESM and Web Streams.

Clearly, ESM and Web Streams are the future. We should support them now and, at some point in time, make a jump to them.

The real questions are "how" and "when."

CJM vs ESM

Current Situation Overview:

stream-json is written as a CJS project for historical reasons—the project is "that old."

It works with native Node streams for the same historical reasons and cannot be used outside of it. Nowadays, we have some alternative runtimes that potentially can support Node streams.

It is also based on stream-chain, which is a CJS project. (The same circular reasons apply).

Considerations and Solutions:

Why do we still use CJS when ESM becomes available?

The main reason: a user can import a CJS module from an ESM module, but the opposite is not true.

Solutions

It leads to two possible solutions:

1. Leave it as is
2. Create a separate ESM version (using a transpiler?)

The former is less complicated and requires fewer tools and less maintenance.

The latest Node versions include experimental support for loading ESMs from CJS modules using require(), which is still behind a flag (as of Node 22): Node.js ECMAScript Modules

Final Note:

We support importing stream-json from ESM using the default Node means. Yet, it precludes using other environments like browsers, deno, bun...

Node streams vs. Web Streams

ESM Limitations and Transition to Web Streams

ESM alone will not solve all problems. We need to support Web Streams. Learn more about Web Streams here. (As of Node 21 Web Streams are no longer experimental).

Structural Challenges

There is a structural problem: stream-json is written in a simple, classic OOP style using inheritance and extending classes provided by Node. Web Streams use a factory pattern where overriding methods are provided as a constructor argument. The class/object composition is different too.

API Changes for Web Streams

Supporting Web Streams will require changes in the API. While stream-json provides factories for every class, it exposes classes themselves too, and they are in use by users.

Handling Node Streams

The other problem is what to do with Node streams. Should we still support them? It is not that difficult to support both types, but how to handle it if they are not available? There is no conditional import. Should we try to import() them asynchronously?

The way forward

What is the way forward?

• Clean break
• A parallel project
• Slowly morphing based on Node changes

Thoughts?

[ShravanSunder]

There's a lot of complexity that comes with bundling cjs and esm modules together and supporting node native and web native projects. Perhaps its better for a parallel project to be started? It would make importing and using the lib much easier.

[uhop]

Supporting two code bases fixing the same bugs and adding the same functionality is also problematic. They drift apart and it is hard to keep them in sync at some point. Unfortunately, I speak from experience.

[sneko]

If it was me, I would keep 1 code base, and only 1 package (having multiple format inside, CJS+ESM). In the past I was dealing with this manually and it was a mess, now there is https://github.com/egoist/tsup that may help you keeping it concise?

2 notes while using your library:

• In my case I'm doing a CLI, and a few libraries I use are compiled ESM only so I have to stay with ESM myself. Using Node v22 could be a solution to avoid a few error messages, but this version is too recent. The users of my library should all upgrade their own Node.js version... which I guess is a nogo for some.

• The stream-json examples show import ... from 'stream-json/Parser';, for ESM it will complain

Cannot find module '/xxxxxxxxxxx/node_modules/stream-json/Parser' imported from /xxxxxxxxxxx/dist/index.js
Did you mean to import "stream-json/Parser.js"?

The easy fix is indeed to use import ... from 'stream-json/Parser.js';. Maybe this could be documented?