Uploading a Request made from a ReadableStream body #425

yutakahirano · 2016-11-28T05:50:55Z

This change enables developers to create a request from a ReadableStream body. fetch() reads data from the body and sends it to the server.

Fixing yutakahirano/fetch-with-streams#66.

annevk

Generally this looks okay, but I'm a little concerned that the first step of https://fetch.spec.whatwg.org/#concept-http-network-or-cache-fetch is not modified. It seems this should directly affect that, no?

annevk · 2016-11-28T09:58:38Z

fetch.bs

@@ -589,6 +589,9 @@ user-agent-defined <a for=header>value</a> for the

 <li><p>A <dfn export for=body id=concept-body-total-bytes>total bytes</dfn> (an
 integer), initially 0.
+
+ <li><p>A <dfn export for=body id=concept-body-replayable>replayable flag</dfn> (a boolean),
+ initially false.


I know this is annoying, and at some point we're going to burn all the flags, but for now we should keep consistency and only set and unset flags and don't say they're booleans.

Also, I think we want to call this cloneable maybe? That is what the operation we want to fail is called.

Replaced boolean/true/false with (none)/set/unset.

We actually clone a body regardless of the flag. So calling it cloneable is confusing, I think.

annevk · 2016-11-28T09:59:29Z

fetch.bs

@@ -3025,6 +3028,11 @@ in addition to <a>HTTP fetch</a> above.
 <a lt="include credential">includes credentials</a>, then return a
 <a>network error</a>.

+ <li><p>If <var>request</var>'s <a for=request>method</a> is not <code>303</code>,


I think you mean to check response's status here, no?

Thanks, fixed.

annevk · 2016-11-28T10:04:01Z

fetch.bs

-typedef (Blob or BufferSource or FormData or URLSearchParams or USVString) BodyInit;
-
-typedef (BodyInit or ReadableStream) ResponseBodyInit;</pre>
+typedef (Blob or BufferSource or FormData or URLSearchParams or USVString or ReadableStream) BodyInit;</pre>


I think we should continue to put USVString last.

annevk · 2016-11-29T08:18:03Z

Okay, so you are saying that request.clone() will continue to work. However, I don't understand why https://fetch.spec.whatwg.org/#concept-http-network-or-cache-fetch should continue to clone. The whole reason we have additional checks is that we can be more conservative there.

annevk · 2016-11-29T08:21:14Z

fetch.bs

@@ -3025,6 +3027,11 @@ in addition to <a>HTTP fetch</a> above.
 <a lt="include credential">includes credentials</a>, then return a
 <a>network error</a>.

+ <li><p>If <var>actualResponse</var>'s <a for=response>status</a> is not <code>303</code>,
+ <var>request</var>'s <a for=request>body</a> is not null, and <var>request</var>'s


annevk · 2016-11-29T08:21:20Z

fetch.bs

@@ -3366,6 +3373,10 @@ steps:
   <li class=XXX><p>Needs testing: multiple `<code>WWW-Authenticate</code>` headers, missing,
   parsing issues.

+   <li><p>If <var>request</var>'s <a for=request>body</a> is not null and <var>request</var>'s


yutakahirano · 2016-11-29T10:22:36Z

However, I don't understand why https://fetch.spec.whatwg.org/#concept-http-network-or-cache-fetch should continue to clone. The whole reason we have additional checks is that we can be more conservative there.

As concept body has the body data as a ReadableStream, we need to clone it to replay uploading regardless of the source data type (String, ArrayBuffer, ReadableStream, ...). There are some options I could think of to handle this issue:

Stop holding the body data as a ReadableStream.
Define a new concept, say "replayable stream", to handle replaying
Keep teeing, but adding a flag to forbid user-visible teeing to allow implementers to omit teeing.

I chose the last option as it looked the easiest.

annevk · 2016-11-29T10:37:15Z

I think we should way more carefully explain that setup then, but I don't like it as much as just being explicit about it as we have done thus far. Maybe 1 is the better solution if that is what implementations actually do.

yutakahirano · 2016-12-05T05:36:15Z

Hmm, 2ab2fce introduces the original object (source) instead of replayable flag. Does that look better to you? With the change we don't need teeing stream for body in http-network-or-cache-fetch, but apparently cloning a request is needed for headers according to the comment. Should I create clone-request-without-teeing-body?

annevk · 2016-12-07T16:44:06Z

Should I create clone-request-without-teeing-body?

Yeah, I think that might be needed. There used to be some handwavy language along the lines of "clone except for body", but maybe that's gone now due to other changes.

annevk · 2016-12-07T16:45:07Z

fetch.bs

@@ -4268,6 +4301,8 @@ typedef (BodyInit or ReadableStream) ResponseBodyInit;</pre>
    <p>If <var>object</var>'s {{Blob/type}}
    attribute is not the empty byte sequence, set <var>Content-Type</var> to its value.

+    <p>Set <var>source</var> to object.


<var>object</var> here and below, no?

annevk · 2016-12-07T16:45:41Z

fetch.bs

+     then return a <a>network error</a>.
+
+     <li>
+      <p>


This <p> seems like a typo.

yutakahirano · 2016-12-09T10:04:39Z

Removed teeing from http-network-or-cache-fetch. PTAL again.

annevk · 2016-12-16T08:34:38Z

fetch.bs

@@ -3093,16 +3093,36 @@ steps:
 <i>authentication-fetch flag</i>.

 <ol>
+ <li><p>Let <var>httpRequest</var> be a new <a for=/>request</a>.


It seems you should just let it be null, no? Since every substep overrides what it is.

annevk · 2016-12-16T08:36:16Z

fetch.bs

-  "<code>error</code>", and the result of <a lt=clone for=request>cloning</a>
-  <var>request</var> otherwise.
+  <p>Otherwise, if <var>request</var>'s <a for=request>body</a> is null, then set
+  <var>httpRequest</var> to a copy of <var>request</var> except for its <a for=request>body</a>.


It seems you can combine this step with the next step. And only if request's body is non-null do the dance of moving the pointer across.

I thought the current structure was easier to read because it saved one nesting level. Don't you think so?

I'm not sure. The fact that the note needs to be duplicated makes me think it's not.

annevk · 2016-12-16T08:38:45Z

Apologies for the delay, I've been traveling/vacationing.

Another thing I forgot to ask for is tests. We're trying to get to a place where all specification changes are accompanied by tests in w3c/web-platform-tests.

annevk · 2016-12-16T08:54:32Z

fetch.bs

+  nullity has already been checked.
+
+  <p class="note no-backref">The <a lt=extract for=BodyInit>extracting</a> operation cannot throw
+  as it was called for the same <a for=body>source</a> before.


Can you combine these two notes into one?

annevk · 2016-12-16T08:56:32Z

fetch.bs

+
+   <li><p>Set <var>request</var>'s <a for=request>body</a> to a new <a for=/>body</a> whose
+   <a for=body>stream</a> is null and whose <a for=body>source</a> is <var>httpRequest</var>'s
+   <a for=request>body</a>'s <a for=body>source</a>.


I think it's worth pointing out here why the request isn't simply cloned.

yutakahirano · 2016-12-19T10:59:13Z

I'll write tests soon...

annevk · 2016-12-19T12:27:34Z

fetch.bs

+    <p class="note no-backref">Here we do not <a for=request>clone</a> <var>request</var> in order
+    to reduce memory consumption. <var>request</var> can be reused with redirects, authentication,
+    and proxy authentication.
+  </ol>


This still seems wrong. Shouldn't we be setting httpRequest's body rather than that of request? Also, what happens to request's body?

The note should probably also move to after the </ol> closing tag. We copy and move body around to preserve memory consumption. And we copy in the first place since it needs to be possible to add headers without affecting request, as request is reused with redirects et al. It seems you dropped part of that note, but we should keep that.

Oops, sorry. Fixed. The intention is, passing the ReadableStream to httpRequests for sending the payload and keeping the source information in request in order to restore the payload if necessary.

annevk · 2016-12-19T12:32:08Z

fetch.bs

@@ -577,7 +577,7 @@ user-agent-defined <a for=header>value</a> for the
 <ul>
 <li>
  <p>A <dfn export for=body id=concept-body-stream>stream</dfn> (a
-  {{ReadableStream}} object).
+  {{ReadableStream}} object or null).


Nit: this should be "null or a ReadableStream object". Maybe now is also a good time to remove the XXX paragraph that follows this? It's been long resolved per yutakahirano/fetch-with-streams#46.

guest271314 · 2016-12-19T21:30:28Z

Does body at body:/* ReadableStream instance */ call .getReader().read() on the value set as to body? If yes, before passing the accumulated value to fetch() call?

annevk · 2016-12-20T07:35:59Z

@guest271314 I appreciate you're interested in learning how this all works, but a better place to ask questions is on IRC (#whatwg, Freenode). Pull requests are meant for reviewing changes to a standard.

yutakahirano · 2017-01-05T07:25:47Z

Can you take a look again?

annevk · 2017-01-05T09:20:00Z

I didn't realize you were waiting for me, apologies. Let me clean up the PR a bit and evaluate where we stand.

Basic test: web-platform-tests/wpt#4362. More tests are expected to be written as part of the implementation effort. Further work: #441. Fixes #88.

annevk · 2017-01-05T09:33:58Z

fetch.bs

+
+  <p class="note no-backref">Here we do not <a for=request>clone</a> <var>request</var> in order
+  to reduce memory consumption. <var>request</var> can be reused with redirects, authentication,
+  and proxy authentication.


This note is still problematic I think compared to the original. I think it should say something like the following:

A request is copied as httpRequest here as we need to be able to add headers to httpRequest and read its body without affecting request. Namely, request can be reused with redirects, authentication, and proxy authentication. We copy rather than clone in order to reduce memory consumption. In case request's body's source is a ReadableStream object, redirects and authentication will end up failing the fetch.

Does that make sense to you?

I'd say "request's body's source is null" or "request's body is made from ReadableStream" insetad of "request's body's source is a ReadableStream object".

Right, yes, that makes sense.

Fetch post-call Always return 415 ？？？？？？？？？？WHY

annevk · 2017-01-05T09:36:33Z

Aside from that note, the other thing I'd like to understand is what the plan is surrounding the contents of these streams. When should we fail upon non-Uint8Array object usage? Is that going to be fixed as part of #441 for both requests and responses?

This simplifies some wording introduced in #425 and aligns it with the recently introduced cloning for service workers. Fixes #1176.

yutakahirano · 2022-05-10T16:29:57Z

Sorry for the long silence.

We, Chrome, did try to prove that this feature is useful, with gRPC/web. But gRPC/web lost its interest to this feature, given we now have WebTransport. As a result, I'm not confident that I can persuade Chromium API owners to ship this feature, given the spec/impl complexity needed for this feature.

FWIW gRPC/web was the only web developer who wanted to use this feature with HTTP/1.1, so I believe we all are now happy to drop the support of this feature on HTTP/1.1.

We have two paths forward:

Get more support from other browser vendors and web developers, and ship this feature.
Give up this feature and remove the logic from the spec.

Thoughts?

bradisbell · 2022-05-10T16:58:08Z

FWIW gRPC/web was the only web developer who wanted to use this feature with HTTP/1.1, so I believe we all are now happy to drop the support of this feature on HTTP/1.1.

I had no idea this was available on HTTP/1.1... we've been needing this for streaming media for a very long time, and yet previous discussions stated that it was locked to HTTP/2. Existing servers, such as Icecast and FFmpeg, do not support HTTP/2 and have no need to. They also do not necessarily need to support HTTPS.

I would strongly urge reconsidering. Please keep HTTP/1.1 streaming request bodies available, and tell us how to use them in Chromium. It does not seem to be available with current builds. If there is some sort of flag we need, what is it? And then, we should go back to all of the threads where this was discussed and inform folks that it is available:

drzraf · 2022-05-10T17:20:24Z

I had no idea this was available on HTTP/1.1... [...] I would strongly urge reconsidering. Please keep HTTP/1.1 streaming request bodies available, and tell us how to use them in Chromium. It does not seem to be available with current builds.

This feature has never been part of a Chrome release. The specs is here, but implementations are missing. Chrome devs told they would implement it, but since it's too complex for Chrome's code base, could the specs please be altered to make it HTTP2+ and then only it may be shipped inside Chrome.

bradisbell · 2022-05-10T17:37:27Z

@drzraf Ok, thank you for that clarification.

ioquatix · 2022-05-10T22:23:05Z

If it's helpful, https://github.com/socketry/async-http supports full bi-directional HTTP/1.1 and HTTP/2 client and server. So you can test with real world code.

By the way, maybe this is chicken and egg problem. Because no browser supports it, and not many people know it's possible, we reach for different tools like WebSocket etc. But frankly as an implementor of HTTP/1 and 2, this is a great disappointment since HTTP itself does support full bi-directional communication and it still remains under-utilized. WebSocket and other solutions are harder to use and harder to implement, and make things more complicated. I'm not a fan of WebSocket despite implementing both client and server - it's a messy spec and it makes HTTP implementation messy.

"We won't implement it because no one is using it" is a self-deprecating argument so I think it's invalid. Not sure what the solution here is, but ultimately I think we should lead with a good specification - I'd definitely hope teams like Node.js are more willing to entertain such a proposal (in terms of a direct implementation of fetch API). I'd certainly be willing to try out a implementation in Ruby that follows an identical semantic model.

davidfowl · 2022-05-11T02:44:09Z

I'm ready to add support for SignalR in .NET, I'd love to chrome and edge to support this. It would be enough to justify doing adding this transport.

yutakahirano · 2022-05-11T05:56:40Z

I had no idea this was available on HTTP/1.1... [...] I would strongly urge reconsidering. Please keep HTTP/1.1 streaming request bodies available, and tell us how to use them in Chromium. It does not seem to be available with current builds.

This feature has never been part of a Chrome release. The specs is here, but implementations are missing. Chrome devs told they would implement it, but since it's too complex for Chrome's code base, could the specs please be altered to make it HTTP2+ and then only it may be shipped inside Chrome.

FTR Chrome ran an origin trial for this feature. Web developers were able to use the feature at that time.

annevk · 2022-05-11T06:48:43Z

@yutakahirano can you please open a new issue to track this? I suppose part of it is covered by #966 but deciding on the fate of this feature seems worthy of its own issue.

yutakahirano · 2022-05-11T07:13:52Z

Sure!

yutakahirano · 2022-05-11T07:39:03Z

Here it is: #1438

yutakahirano force-pushed the upload-stream branch from 079a0be to 82d0b86 Compare November 28, 2016 05:52

annevk requested changes Nov 28, 2016

View reviewed changes

annevk requested changes Nov 29, 2016

View reviewed changes

annevk reviewed Dec 7, 2016

View reviewed changes

annevk reviewed Dec 16, 2016

View reviewed changes

annevk mentioned this pull request Dec 19, 2016

Setting body as ReadableStream at POST request #439

Closed

yutakahirano mentioned this pull request Dec 19, 2016

Add a fetch test for upload streaming web-platform-tests/wpt#4362

Merged

annevk reviewed Dec 19, 2016

View reviewed changes

yutakahirano mentioned this pull request Dec 20, 2016

Define "to read a request" more specifically #441

Closed

Add upload streams (a Request with a ReadableStream body)

0cef38e

Basic test: web-platform-tests/wpt#4362. More tests are expected to be written as part of the implementation effort. Further work: #441. Fixes #88.

annevk force-pushed the upload-stream branch from e57203c to 0cef38e Compare January 5, 2017 09:26

annevk reviewed Jan 5, 2017

View reviewed changes

drzraf mentioned this pull request May 19, 2020

Openpgp.message.fromBinary doesn't work with Nodejs.ReadableStream openpgpjs/openpgpjs#994

Closed

an0ndev mentioned this pull request May 25, 2020

Request should allow WritableStream as body #378

Closed

nwtgck mentioned this pull request May 28, 2020

Chunked (pseudo-stream) encryption & upload nwtgck/piping-ui-web#433

Closed

This was referenced Jun 13, 2020

Investigage full-duplex request/response streaming servo/servo#26905

Open

Make streaming upload work. servo/servo#26904

Closed

whatwg deleted a comment from BingyuLiu Feb 2, 2021

annevk added a commit that referenced this pull request Feb 19, 2021

Give all bodies a stream

2fa49c0

This simplifies some wording introduced in #425 and aligns it with the recently introduced cloning for service workers. Fixes #1176.

annevk mentioned this pull request Feb 19, 2021

Give all bodies a stream #1177

Merged

annevk added a commit that referenced this pull request Feb 24, 2021

Give all bodies a stream

0f29092

This simplifies some wording introduced in #425 and aligns it with the recently introduced cloning for service workers. Fixes #1176.

nwtgck mentioned this pull request May 9, 2021

Screensharing firefox “text/plain” is not supported. nwtgck/piping-server#472

Closed

drzraf mentioned this pull request Oct 12, 2021

WIP: async read function by default flowjs/flow.js#368

Open

ronag mentioned this pull request May 11, 2022

https://github.com/whatwg/fetch/pull/425/files nodejs/undici#1432

Closed

annevk added the topic: streams label May 11, 2022

yutakahirano mentioned this pull request May 11, 2022

Streaming upload #1438

Closed

yutakahirano mentioned this pull request Jul 5, 2022

fetch streaming upload mozilla/standards-positions#663

Closed

anthumchris mentioned this pull request Dec 4, 2022

Add Example for File Upload anthumchris/fetch-progress-indicators#6

Open

imaksp mentioned this pull request Apr 10, 2024

reportProgress not working angular/angular#53650

Open

dandclark mentioned this pull request Oct 7, 2024

Fetch Request Streaming web-platform-tests/interop#820

Open

Uploading a Request made from a ReadableStream body #425

Uploading a Request made from a ReadableStream body #425

Conversation

yutakahirano commented Nov 28, 2016

annevk left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annevk commented Nov 29, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yutakahirano commented Nov 29, 2016 • edited Loading

annevk commented Nov 29, 2016

yutakahirano commented Dec 5, 2016

annevk commented Dec 7, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yutakahirano commented Dec 9, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annevk commented Dec 16, 2016

annevk Dec 16, 2016 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

yutakahirano commented Dec 19, 2016

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

guest271314 commented Dec 19, 2016

annevk commented Dec 20, 2016

yutakahirano commented Jan 5, 2017

annevk commented Jan 5, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

annevk commented Jan 5, 2017

yutakahirano commented May 10, 2022

bradisbell commented May 10, 2022 • edited Loading

drzraf commented May 10, 2022 • edited Loading

bradisbell commented May 10, 2022

ioquatix commented May 10, 2022 • edited Loading

davidfowl commented May 11, 2022

yutakahirano commented May 11, 2022

annevk commented May 11, 2022

yutakahirano commented May 11, 2022

yutakahirano commented May 11, 2022

yutakahirano commented Nov 29, 2016 •

edited

Loading

annevk Dec 16, 2016 •

edited

Loading

bradisbell commented May 10, 2022 •

edited

Loading

drzraf commented May 10, 2022 •

edited

Loading

ioquatix commented May 10, 2022 •

edited

Loading