Use in a streaming library for nodejs and browsers

[traviscollins]

I have a project that has a library of streaming classes that are intended to be used both in NodeJS runtimes and in browsers. I'd like to use web-streams-polyfill in the browser.

The main problem is that web-streams-polyfill requires the use of "web-streams-polyfill/es2018" (or other) as an import reference. Where as NodeJS provides the "stream" import reference. This means that the classes in my library are not compilable in a browser context (complains about "stream" dependency not found).

I've got a few hacky workaround ideas, but wondered if there's an immediately obvious solution that I'm not seeing.

[MattiasBuelens]

The polyfill currently doesn't make any assumptions about its runtime environment besides those mentioned in the README, so it doesn't try to integrate with Node's own implementation from node:stream/web.

You could work around this in your own code base, assuming you can use dynamic import() with top-level await:

// streams.js
let streams;
try {
  streams = await import("node:stream/web");
} catch {
  streams = await import("web-streams-polyfill/es2018");
}
const { ReadableStream, WritableStream, TransformStream } = streams;
export { ReadableStream, WritableStream, TransformStream };

// main.js
import { ReadableStream } from "./streams.js";
let readable = new ReadableStream({ /* ... */ });

For version 4.0, I'm going to migrate to conditional exports to support import and require(), see package.json. If there's enough demand, I could investigate whether we should add a conditional export for Node too, which can then re-export Node's own implementation if available. 🤔

[jimmywarting]

extend that ☝️ to browser stream as well

// streams.js
let streams;
if (globalThis.WritableStream && globalThis.ReadableStream && globalThis.TransformStream) {
  streams = globalThis
} else {
  try {
    streams = await import("node:stream/web");
  } catch {
    streams = await import("web-streams-polyfill/es2018");
  }
}
const { ReadableStream, WritableStream, TransformStream } = streams;
export { ReadableStream, WritableStream, TransformStream };

// main.js
import { ReadableStream } from "./streams.js";
let readable = new ReadableStream({ /* ... */ });

[traviscollins]

The package.json specification has a "browser" property that allows you specify different entry modules for browsers vs nodejs. Could leave the main module empty, and just populate the browser module.

https://docs.npmjs.com/cli/v8/configuring-npm/package-json

[fellowseb]

How would you proceed without the support for top-level await ?

[poonam-221]

I am getting Unable to resolve "web-streams-polyfill/polyfill". Node_modules contains it. I have latest version of web-streams-polyfill. package.json has "web-streams-polyfill": "^4.0.0" in dependencies. I deleted .expo, node_modules etc multiple times. I am using langchain and that's where I am getting ReadableStream error so need this polyfill. Please help.

[MattiasBuelens]

@poonam-221 Please create a separate issue for your problem.

It would also help if you could provide a minimal reproducible example. In your case, a standalone Git repository that demonstrates the problem would be ideal.