Update TransformStream API & misc. fixups #519

isonmad · 2016-09-16T23:26:10Z

First pass at issue #518 to align with the other stream APIs. Ended up also addressing #498, #331, and #185.

transformStream._error does not exist. This codepath really needs a test.

The conditional here is weird.

Align the TransformStream API to resemble ReadableStream more closely. Issue whatwg#518.

Align with the WritableStream API. This resolves issues whatwg#518, whatwg#498, whatwg#331, and whatwg#185.

Return a promise with the same semantics as WritableStream's underlyingSink.close(). This also resolves an old comment, as well as issue whatwg#518. Consequently, calling controller.close() after flush() is called now throws a TypeError.

ricea · 2016-09-20T04:17:49Z

If I understand correctly, this doesn't enforce any ordering guarantees between the start/transform/flush methods. I would like to have the guarantee that transform() is not called until start()'s Promise resolves (and not at all if it rejects), and flush() is not called until transform()'s Promise resolves.

The reason is that if transform() is called before the Promise returned by start() resolves, then a call to enqueue() by transform() could end up happening before before a call to enqueue() by start(). Similarly with transform() and flush().

It's possible to enforce the ordering within the transformer, but that seems likely to cause a steep learning curve as every developer rediscovers the issue.

I can file a separate issue for this if you like.

tyoshino

Thanks for the patch

tyoshino · 2016-09-20T07:19:40Z

reference-implementation/lib/transform-stream.js

-    }
+    const flushPromise = PromiseInvokeOrNoop(transformStream._transformer, 'flush', [transformStream._controller]);
+    // Return a promise that is fulfilled with undefined on success.
+    return flushPromise.then(() => TransformStreamCloseReadableInternal(transformStream),


What if controller.error() is invoked inside flush() but flush() resolved? I think this path should perform TransformStreamCloseReadableInternal() when the transform stream is neither errored nor closed.

You're right, that should be fixed, though I'm tempted to make controller.error() throw after flush() is called, like with controller.close().

Otherwise, what should happen if controller.error() is called with one error, and flush throws or rejects the promise with a different error? What should happen if controller.close() is called, but then the flushPromise is rejected? It seemed simpler to give flush only a single way of signaling success or failure. You already can't signal error() or close() twice after all.

I understand your point. But we could also say that close() becoming disallowed in flush() would make the API a little complicated. Regarding the priority concern, I think we should make the transform stream accept the signal which arrived to it earliest. That said, this kind of rule also complicates the API.

Another question I'm not yet fully sure about is whether the resolution of flushPromise which you've changed to be propagated to writer.close() and readable's final state should always match or not. If close() / error() are allowed inside flush(), we can e.g. close() readable side while making writer.close() rejected. My gut feeling is basically they should match each other, but it seems worth being discussed a bit.

tyoshino · 2016-09-20T07:28:03Z

reference-implementation/lib/transform-stream.js

@@ -16,6 +17,17 @@ function TransformStreamCloseReadable(transformStream) {
    throw new TypeError('Readable side is already closed');
  }

+  if (transformStream._writableDone === true) {
+    throw new TypeError('TransformStream is already closing');
+  }


Did you try to limit the way to close the readable side to only by resolving flush() once flush() is invoked? What's the motivation for that?

Having two redundant ways to perform the same action seemed dubious (especially if, as in your other comment, the result is potentially contradictory, instead of redundant). It's already an error to call close() twice, this seemed roughly analogous to that.

isonmad · 2016-09-20T22:57:42Z

I would like to have the guarantee that transform() is not called until start()'s Promise resolves (and not at all if it rejects), and flush() is not called until transform()'s Promise resolves.

start() doesn't return a promise yet, but adding one with that invariant makes sense. But flush() is only called from sink.close(), and that's already guaranteed not to be called until the last sink.write() promise resolves, which is only resolved when a transform() promise resolves, if I understand it all correctly. There's already an assert(transformStream._transforming === false); before the flush() call to that effect.

Now all 3 transformer methods are promise-ified.

Disregard previous commit. sink.start and source.start are called synchronously so this should too. Much simpler.

The conditional is always true, so this always throws. Is there a point in altering the state of an object when throwing an exception from its constructor anyway?

This matches behavior for readable and writable streams.

And allow both.

Per the spec, ReadableStreamDefaultControllerGetDesiredSize is nothrow. Nothing can go wrong with just adding up numbers.

Before, it was trying to error the readable end, when it had already errored itself.

tyoshino · 2016-09-29T12:48:03Z

Sorry for delay. I'll review this soon.

tyoshino

lgtm once the 2 comments are addressed. thanks so much.

tyoshino · 2016-10-03T05:44:03Z

reference-implementation/lib/transform-stream.js

-  if (transformStream._transformer.start === undefined) {
-    return;
-  }
+  const p = PromiseInvokeOrNoop(transformStream._transformer, 'transform', [chunk, transformStream._controller]);


s/p/transformPromise/

tyoshino · 2016-10-03T08:04:12Z

reference-implementation/test/transform-stream.js

+  writer.write('a');
+  writer.close().then(() => {
+    t.ok(flushDone, 'flushPromise resolved');
+    t.end();


unnecessary as you have t.plan()

tyoshino · 2016-10-05T06:26:28Z

lgtm. thanks!

tyoshino · 2016-10-05T06:30:17Z

I'd like to add your name to the acknowledgement section of the spec. Do you want to use isonmad or some other name?

I'm going to squash the commits and merge it. If you want to write the commit summary for the squashed commit, please let me know. I can also write it for you if you want.

This function is just one more abstract operation now and doesn't need its own separate category.

domenic · 2016-10-10T20:59:29Z

Just wanted to say thank you @isonmad for all this work! It's really excellent to get this kind of contribution.

I guess we can merge this now with "isonmad" in the acknowledgments, but do feel free to let us know what would be better!

isonmad · 2016-10-10T21:59:27Z

That works fine. If it has to all be one monolithic commit, I guess the commit message would be

move TransformStream to promises based api (#519)

Fix a bunch of bugs along the way and add tests for them.

Closes #518, #498, #331, and #185.

tyoshino · 2016-10-12T05:22:25Z

Thanks so much for your work, @isonmad. I've added your name to the acknowledgement section in 294d720

I'm going to make some follow up commits.

Follow up for #519 As we error the writable stream in TransformStreamErrorInternal, we don't need to take care of the promise returned from the underlying sink write().

tyoshino · 2016-10-12T05:39:39Z

reference-implementation/lib/transform-stream.js


    transformStream._chunkPending = true;
    transformStream._chunk = chunk;

    const promise = new Promise(resolve => {
      transformStream._resolveWrite = resolve;
+      transformStream._rejectWrite = resolve;


this should be set to the rejection callback of the promise

tyoshino · 2016-10-12T05:40:51Z

reference-implementation/lib/transform-stream.js

-  if (transformStream._resolveWriter !== undefined) {
-    transformStream._resolveWriter(undefined);
+  if (transformStream._rejectWriter !== undefined) {
+    transformStream._rejectWriter(e);


s/rejectWriter/rejectWrite/

anyway, I just noticed that this is unnecessary since we call _writableController.error() at L118. I'm going to just remove this in #533

Follow up for #519 As we error the writable stream in TransformStreamErrorInternal, we don't need to take care of the promise returned from the underlying sink write().

transform-stream: fix typo

269a970

isonmad force-pushed the transform1 branch 9 times, most recently from 40c3948 to da6fba2 Compare September 18, 2016 18:27

isonmad added 11 commits September 19, 2016 02:06

add transformstream._storedError

9d9474f

fix thrown error in transformstream enqueue

0faf0e1

transformStream._error does not exist. This codepath really needs a test.

add TransformStreamErrorIfNeeded

dbf1fd5

Have TransformStreamSink.close always throw on error

376701c

The conditional here is weird.

add TransformStreamDefaultController

feafe52

Align the TransformStream API to resemble ReadableStream more closely. Issue whatwg#518.

test TransformStream error()

7c1025a

TransformStream: use InvokeOrNoop

7b6313f

add TransformStreamResolveWrite

a99e4cd

Return a Promise from TransformStream transform()

2fd0945

Align with the WritableStream API. This resolves issues whatwg#518, whatwg#498, whatwg#331, and whatwg#185.

add TransformStreamCloseReadableInternal

27bef29

isonmad force-pushed the transform1 branch from da6fba2 to a715e9e Compare September 19, 2016 03:02

tyoshino reviewed Sep 20, 2016

View reviewed changes

isonmad added 5 commits September 21, 2016 00:12

add TransformStreamAdd{Readable,Writable}Controller

ccb6dfe

TransformStream: return promise from transformer.start()

cad1cc1

Now all 3 transformer methods are promise-ified.

TransformStream: call transformer.start synchronously

88c5a7d

Disregard previous commit. sink.start and source.start are called synchronously so this should too. Much simpler.

remove weird conditional throw in TransformStream constructor

f18010d

The conditional is always true, so this always throws. Is there a point in altering the state of an object when throwing an exception from its constructor anyway?

test synchronous transformer.start failure

464df6c

This matches behavior for readable and writable streams.

isonmad added 4 commits September 22, 2016 01:37

TransformStream: account for close() and error() calls in flush

36944ed

And allow both.

drop superfluous try-catch in TransformStream enqueue

b73b21e

Per the spec, ReadableStreamDefaultControllerGetDesiredSize is nothrow. Nothing can go wrong with just adding up numbers.

TransformStream: handle readableStrategy.size() errors

2ca4d03

Before, it was trying to error the readable end, when it had already errored itself.

handle badly behaving readableStrategy.size

ccc0a1c

tyoshino reviewed Oct 3, 2016

View reviewed changes

isonmad added 4 commits October 4, 2016 18:41

fix variable name

04d6fe0

drop t.end

db49655

assert controller objects are present

c3902c6

fix comment

134f089

domenic added the transform streams label Oct 7, 2016

isonmad added 3 commits October 10, 2016 20:38

add _rejectWrite

40c6327

fixup comments

6b2649e

This function is just one more abstract operation now and doesn't need its own separate category.

move internal op next to other abstract ops

7cc3787

make .readable and .writable readonly properties

7c40ebc

domenic mentioned this pull request Oct 12, 2016

Port some TransformStream tests to wpt #531

Merged

domenic assigned tyoshino and ricea Oct 12, 2016

tyoshino merged commit a957cc5 into whatwg:master Oct 12, 2016

tyoshino mentioned this pull request Oct 12, 2016

No need to reject writePromise in TransformStreamErrorInternal #533

Merged

tyoshino reviewed Oct 12, 2016

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Update TransformStream API & misc. fixups #519

Update TransformStream API & misc. fixups #519

isonmad commented Sep 16, 2016 •

edited

Loading

ricea commented Sep 20, 2016

tyoshino left a comment

tyoshino Sep 20, 2016

isonmad Sep 20, 2016 •

edited

Loading

tyoshino Oct 3, 2016

tyoshino Sep 20, 2016

isonmad Sep 20, 2016

isonmad commented Sep 20, 2016 •

edited

Loading

tyoshino commented Sep 29, 2016

tyoshino left a comment

tyoshino Oct 3, 2016

tyoshino Oct 3, 2016 •

edited

Loading

tyoshino commented Oct 5, 2016

tyoshino commented Oct 5, 2016

domenic commented Oct 10, 2016

isonmad commented Oct 10, 2016

tyoshino commented Oct 12, 2016

tyoshino Oct 12, 2016

tyoshino Oct 12, 2016

Update TransformStream API & misc. fixups #519

Update TransformStream API & misc. fixups #519

Conversation

isonmad commented Sep 16, 2016 • edited Loading

ricea commented Sep 20, 2016

tyoshino left a comment

Choose a reason for hiding this comment

tyoshino Sep 20, 2016

Choose a reason for hiding this comment

isonmad Sep 20, 2016 • edited Loading

Choose a reason for hiding this comment

tyoshino Oct 3, 2016

Choose a reason for hiding this comment

tyoshino Sep 20, 2016

Choose a reason for hiding this comment

isonmad Sep 20, 2016

Choose a reason for hiding this comment

isonmad commented Sep 20, 2016 • edited Loading

tyoshino commented Sep 29, 2016

tyoshino left a comment

Choose a reason for hiding this comment

tyoshino Oct 3, 2016

Choose a reason for hiding this comment

tyoshino Oct 3, 2016 • edited Loading

Choose a reason for hiding this comment

tyoshino commented Oct 5, 2016

tyoshino commented Oct 5, 2016

domenic commented Oct 10, 2016

isonmad commented Oct 10, 2016

tyoshino commented Oct 12, 2016

tyoshino Oct 12, 2016

Choose a reason for hiding this comment

tyoshino Oct 12, 2016

Choose a reason for hiding this comment

isonmad commented Sep 16, 2016 •

edited

Loading

isonmad Sep 20, 2016 •

edited

Loading

isonmad commented Sep 20, 2016 •

edited

Loading

tyoshino Oct 3, 2016 •

edited

Loading