WebStream API support

[martinheidegger]

This has been sort of worked on in #140 but then was continued in another branch #140 (comment).

Is the WebStream API, that is published in FF & Chrome, like getReader() etc. too experimental for node-fetch?

[jimmywarting]

Not sure it would be useful in a nodejs enviorment where everything is written with node streams.
the WebStream api isn't small and would add quite alot to node-fetch dependency.

There is a grate package called node-web-streams for conversion between those two

const webStreams = require('node-web-streams');
const readableStream = webStreams.toWebReadableStream(nodeReadable);

if we where to implement ReadableStream in node-fetch I'm guessing most ppl would just to the opposite and try to convert it back to a node stream

It would be a breaking change since Response.prototype.body is now a node stream and some are already using it.

[ThisIsMissEm]

I've recently been implementing the cloudflare-workers API as a local server for development, and support for WebStreams is something we need. Would it be possible to support passing in a set Stream constructors for node-fetch to use?

That way if people want the web-streams API, they can opt-in to it.

[TimothyGu]

@ThisIsMissEm There are nontrivial differences in the stream.Readable API vs. the web ReadableStream API, and using the constructors of the two classes the same way unfortunately doesn't work. See some examples for Web Streams vs. Node.js Streams.

There have some exciting developments over on the Node.js side (nodejs/node#22352), and if Web Streams eventually get in Node core we can certainly add support for it.

(Does this answer your question or am I missing something you are trying to say?)

[ThisIsMissEm]

Yeah, we realised this was the case, so we've forked node-fetch to use ReadableStream API. You can see the start of our attempt here: master...titel-media:whatwg-readablestreams

[ThisIsMissEm]

Whilst my usage of node-fetch is for reimplementing the API of CloudFlare Workers, I think it’s worth noting that several modules that depend on node-fetch claim that they expose the whatwg-fetch spec, which isn’t true, as that’d mean they expose whatwg streams; They’ve possibly missed the note about difference in stream APIs

[ThisIsMissEm]

@TimothyGu We actually have pretty much full tests passing — I removed a bunch of stuff and can do a PR, but in order for my team to move forward we'll be publishing a scoped fork. It's up to you if you want to take my changes into node-fetch, I'm happy to have a fork for now.

[jimmywarting]

Found out something that is going to make this cross browser/node chunk reading much simpler! Async iterator!!!

Node implemented Symbol.asyncIterator into there stream specification (v10) and whatwg are planing on adding them too, until then here is a polyfill for the browser:

if (!ReadableStream.prototype[Symbol.asyncIterator]) {
  ReadableStream.prototype[Symbol.asyncIterator] = async function* () {
    const reader = this.getReader()
    while (1) {
      const r = await reader.read()
      if (r.done) return value
      yield r.value
    }
  }
}

now what you can do in both the browser and Node is:

const res = await fetch('https://httpbin.org/bytes/500000')
for await (const chunk of res.body) {
  console.log(chunk)
}

Both yields a Uint8Array array (well, a buffer in node, but they are inherited by Uint8array) so you can treat them the same

Hopefully this will make the WebStream api for node-fetch more irrelevant
(still think the node-web-streams is to much as a dependency)

---

this is something i would very much like to see in the readme 😛

[TimothyGu]

@jimmywarting Unfortunately, async iterators don't provide a full replacement for streams with regards to things like backpressure. This is obviously a step in the right direction but there is still value in exposing Web streams.

For context, a PR that adds async iterators to web streams is whatwg/streams#950.

[ThisIsMissEm]

@jimmywarting wow! That makes it SO much nicer than using the reader / while loop interface.

[ThisIsMissEm]

@TimothyGu wouldn't backpressure with async iterator just be handled via something like:

const res = await fetch('https://httpbin.org/bytes/500000')
for await (const chunk of res.body) {
  await doSomework(chunk)
}

So because we're delaying in each iteration backpressure applies?

[TimothyGu]

Hm, I guess I was more referring to the highWatermark-based backpressure. That does indeed work, yes.

[ThisIsMissEm]

Okay, so, an update from my end: We're publishing a fork with ReadableStreams as @titelmedia/node-fetch, such that we can use it in our Cloudflare Worker development server. It's up to you @TimothyGu if you choose to adopt the changes I've made.

[jimmywarting]

Think you should have highlighted the differences between your fork and node-fetch in the readme a little better

[ThisIsMissEm]

Oh, yes.. I forgot about that. The package description has it though.

[bitinn]

@TimothyGu not sure your take on this but the nodejs core web stream PR didn't land; it seems the only viable solution is to use https://github.com/MattiasBuelens/web-streams-polyfill for now.

[bitinn]

Some good news:

https://github.com/nodejs/whatwg-stream

[jimmywarting]

it might be ok to leave them as a buffer internally, as long as res.body getter is guaranteed to be a stream (the debate is then whether it should be a web stream or nodejs stream...) - #547 (comment)

So what is the verdict? should we implement web-streams-polyfill for v3?

I guessing some ppl will complain about the switch from node stream to whatwg streams

[bitinn]

@jimmywarting don't think there is a verdict yet, stream is a different spec which I believe isn't quite finished yet, and browser support isn't quite on-par the last time I checked.

I recently thought about what if Fetch options allow users to pass in a stream implementation (node or web), and use that instead, just like how we handled AbortController support.

And since there is a lack of man-power on this issue, another likely outcome is we wait for nodejs to implement it, which will be a long while.

See also: MattiasBuelens/web-streams-polyfill#1

[jimmywarting]

I kinda feel like it's the right step forward to implement whatwg streams (to be isomorphic compatible cross node and browser) while at the same time i also feel like node-fetch is not quite ready for it yet.

here is some other ideas i thought of to keep the users that still uses node-streams events and pipes happy. We could implement a non standard function body.stream() that returns the internal node stream and let body.body be a whatwg stream

Or make some utility function something like

const asNodeStream = require('fetch/as-node-stream') // or
const { asNodeStream } = require('fetch')

fetch(url).then(res => {
  asNodeStream(res.body) // returns the internal body stream
   .pipe(dest)
})

Another idea is the use of symbol.

const { internalNodeStreamSymbol } = require('fetch')

fetch(url).then(res => {
  res[internalNodeStreamSymbol].pipe(dest)
})

[josephrocca]

@ThisIsMissEm I can't thank you enough for sharing that fork! After an hour or two of trying to mash polyfills together unsuccessfully I thought I was going to have to implement something myself.

(sorry for the noise)

[loynoir]

Recently I found https://github.com/nodejs/undici

• whatwg readablestream
• have getReader
• but need node16 to have stream/web, AbortController, ...

[jimmywarting]

We got plans to bring in whatwg streams polyfill and change the response body to whatwg streams for v4, but it's going to be a breaking change... would like to fix some issues before making that release

[RangerMauve]

I've been having great success with web-streams-polygill inside the make-fetch library.

Wasn't too hard to migrate to it from node streams.

[BasixKOR]

By breaking change, do you mean that v4 no longer supports Node.js stream? I'm a bit confused.