-
Notifications
You must be signed in to change notification settings - Fork 30.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
stream-whatwg: add whatwg streams #22352
Changes from all commits
File filter
Filter by extension
Conversations
Jump to
Diff view
Diff view
There are no files selected for viewing
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -24,6 +24,17 @@ | |
|
||
'openssl_fips%': '', | ||
|
||
'v8_extra_library_files': [ | ||
'./lib/v8_extras/ByteLengthQueuingStrategy.js', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How are these files consumed by the build? I see that gypfiles in deps/v8 references There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. That's exactly what is happening. The V8 build compiles the v8_extras in. They are not loaded the same way as every other thing in core. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. can chakra not just put these files somewhere and eval them when a context is created? we do plenty of stuff elsewhere that chakra has to shim. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Yes, chakra shims a lot of things, but so far it has been native APIs (or js APIs). Not side effects of the build system. Sure, it can be done, but I'd much prefer not to. Personally I also feel like this makes the build of node more complex to reason about, it is that much less of a dependency tree and more of a dependency graph. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. If you wanted to have these JS files live in There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @MSLaguana are you saying to just put the list in node.gyp instead of common.gypi? There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not the list, but the thing that consumes the list. The bit that takes the list of filenames and processes it to the point where the relevant build artifact is produced. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Looking through the implementation, I'm not sure which aspects of it would be infeasible to implement as a normal module. Doing so would make it significantly easier to polyfill, would make it more likely that it could just be dropped in to There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. v8 consumes the list and turns it into a c++ file that is linked internally to the engine. chakrashim would need to do something similar, calling those extras functions when new contexts are created. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. @jasnell because its a lot of stuff to rewrite (i don't have the time to do that) and i'd like to be able to collaborate with chromium by keeping at least similar implementations. readable-stream can use any of the polyfills on npm or even vendor the reference implementation from the streams repo. |
||
'./lib/v8_extras/CommonOperations.js', | ||
'./lib/v8_extras/CommonStrings.js', | ||
'./lib/v8_extras/CountQueuingStragety.js', | ||
'./lib/v8_extras/ReadableStream.js', | ||
'./lib/v8_extras/SimpleQueue.js', | ||
'./lib/v8_extras/TransformStream.js', | ||
'./lib/v8_extras/WritableStream.js', | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Given that this is a straight up copy of the chromium code, it may be better to have this in |
||
], | ||
|
||
# Default to -O0 for debug builds. | ||
'v8_optimized_debug%': 0, | ||
|
||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -341,6 +341,23 @@ reader.pipe(writer); | |
reader.unpipe(writer); | ||
``` | ||
|
||
##### writable.acquireStandardStream() | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Not sure about calling these "...StandardStream"... Don't really have a better suggestion right now tho There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. This will definitely need more documentation about the new objects There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. Another thing to keep in mind is that the const { toWHATWGStream } = require('stream')
const myStream = toWHATWGStream(fs.createWriteStream()) This should allow it to work with any object that implements the right methods and events. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more.
i don't really understand this. a polyfill of whatwg streams shouldn't have a concept of node streams. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. The point is this can be done in a way where this isn't even a problem by not extending the prototype. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. sure it can, but I'm trying to design a good API, not a easy-for-some-npm-module-to-not-worry-about API. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. "good" is entirely subjective here. Others will likely have very different opinions of what is "good". The "readable-streams" module is not just "some-npm-module", it's the most downloaded module in the npm ecosystem and is part of the Node.js project managed by the Streams WG. Given that, I'm very concerned about implementation decisions that directly impact it. Yes, we're able to add features to Node.js core before they propagate down to the module, but we must do so carefully and deliberately, weighing each choice on the impact it may have. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. I'm hopeful we can come up with something that isn't importing separate functions. I want to have as little friction as possible for people wanting to use whatwg streams, as there are quite a lot of locations where the interop needs to happen. I'm willing to continue bikeshedding this for as long as we need to for something that works for everyone, as it would seem we both have constraints here. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. There are some relevant comments here that are being buried a bit after rebasing... just linking them here so that they don't get lost in the conversation: 1ca4477#r211124305 and 1ca4477#r211131273 |
||
<!-- YAML | ||
added: REPLACEME | ||
--> | ||
|
||
> Stability: 1 - Experimental | ||
|
||
* Returns: {WritableStream} to feed this stream. | ||
|
||
```js | ||
const fs = require('fs'); | ||
|
||
const stream = fs.createWriteStream('file').acquireStandardStream(); | ||
const writer = stream.getWriter(); | ||
writer.write('hi!'); | ||
``` | ||
|
||
##### writable.cork() | ||
<!-- YAML | ||
added: v0.11.2 | ||
|
@@ -827,6 +844,22 @@ If both `'readable'` and [`'data'`][] are used at the same time, `'readable'` | |
takes precedence in controlling the flow, i.e. `'data'` will be emitted | ||
only when [`stream.read()`][stream-read] is called. | ||
|
||
##### readable.acquireStandardStream() | ||
<!-- YAML | ||
added: REPLACEME | ||
--> | ||
|
||
> Stability: 1 - Experimental | ||
|
||
* Returns: {ReadableStream} to fully consume the stream. | ||
|
||
```js | ||
const fs = require('fs'); | ||
|
||
const stream = fs.createReadStream('file').acquireStandardStream(); | ||
const reader = stream.getReader(); | ||
``` | ||
|
||
##### readable.destroy([error]) | ||
<!-- YAML | ||
added: v8.0.0 | ||
|
@@ -1266,6 +1299,26 @@ Examples of `Duplex` streams include: | |
* [zlib streams][zlib] | ||
* [crypto streams][crypto] | ||
|
||
##### duplex.acquireStandardStream | ||
<!-- YAML | ||
added: REPLACEME | ||
--> | ||
|
||
> Stability: 1 - Experimental | ||
|
||
* Returns {Object} | ||
* `readable` {ReadableStream} | ||
* `writable` {WritableStream} | ||
|
||
Creates a WHATWG stream pair to represent this duplex stream. | ||
|
||
```js | ||
const stream = getDuplexSomehow(); | ||
const { readable, writable } = stream.acquireStandardStream(); | ||
readable.getReader(); | ||
writable.getWriter(); | ||
``` | ||
|
||
#### Class: stream.Transform | ||
<!-- YAML | ||
added: v0.9.4 | ||
|
@@ -2200,6 +2253,15 @@ by [`stream._transform()`][stream-_transform]. The `'end'` event is emitted | |
after all data has been output, which occurs after the callback in | ||
[`transform._flush()`][stream-_flush] has been called. | ||
|
||
##### transform.acquireStandardStream() | ||
<!-- YAML | ||
added: REPLACEME | ||
--> | ||
|
||
> Stability: 1 - Experimental | ||
|
||
* Returns: {TransformStream} | ||
|
||
#### transform.\_flush(callback) | ||
|
||
* `callback` {Function} A callback function (optionally with an error | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
|
@@ -591,6 +591,33 @@ Writable.prototype.end = function(chunk, encoding, cb) { | |
return this; | ||
}; | ||
|
||
Writable.prototype.acquireStandardStream = function() { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. How are errors propagated here? There's nothing listening to the error event. |
||
internalUtil.emitExperimentalWarning('Writable.acquireStandardStream'); | ||
return new WritableStream({ | ||
start: (controller) => { | ||
this.once('error', (e) => controller.error(e)); | ||
}, | ||
write: (chunk) => { | ||
return new Promise((resolve) => { | ||
const underHighWaterMark = this.write(chunk); | ||
if (!underHighWaterMark) { | ||
this.once('drain', resolve); | ||
} else { | ||
resolve(); | ||
} | ||
}); | ||
}, | ||
close: (controller) => { | ||
this.end(); | ||
}, | ||
abort: (reason) => { | ||
this.destroy(reason); | ||
}, | ||
}, { | ||
highWaterMark: this.writableHighWaterMark, | ||
}); | ||
}; | ||
|
||
Object.defineProperty(Writable.prototype, 'writableLength', { | ||
// making it explicit this property is not enumerable | ||
// because otherwise some prototype manipulation in | ||
|
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,13 @@ | ||
/* eslint-disable no-restricted-globals */ | ||
|
||
const globals = new Set(Object.getOwnPropertyNames(global)); | ||
globals.delete('undefined'); | ||
|
||
module.exports = { | ||
extends: ['../.eslintrc.yaml'], | ||
rules: { | ||
'strict': ['error', 'function'], | ||
'no-restricted-syntax': 'off', | ||
'no-restricted-globals': [2, ...globals], | ||
}, | ||
}; |
Original file line number | Diff line number | Diff line change |
---|---|---|
@@ -0,0 +1,55 @@ | ||
// Copyright 2015 The Chromium Authors. All rights reserved. | ||
// | ||
// Redistribution and use in source and binary forms, with or without | ||
// modification, are permitted provided that the following conditions are | ||
// met: | ||
// | ||
// * Redistributions of source code must retain the above copyright | ||
// notice, this list of conditions and the following disclaimer. | ||
// * Redistributions in binary form must reproduce the above | ||
// copyright notice, this list of conditions and the following disclaimer | ||
// in the documentation and/or other materials provided with the | ||
// distribution. | ||
// * Neither the name of Google Inc. nor the names of its | ||
// contributors may be used to endorse or promote products derived from | ||
// this software without specific prior written permission. | ||
// | ||
// THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS | ||
// "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT | ||
// LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR | ||
// A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT | ||
// OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, | ||
// SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT | ||
// LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, | ||
// DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY | ||
// THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT | ||
// (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE | ||
// OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. | ||
|
||
// eslint-disable-next-line no-unused-vars | ||
(function(global, binding, v8) { | ||
'use strict'; | ||
|
||
const defineProperty = global.Object.defineProperty; | ||
|
||
class ByteLengthQueuingStrategy { | ||
constructor(options) { | ||
defineProperty(this, 'highWaterMark', { | ||
value: options.highWaterMark, | ||
enumerable: true, | ||
configurable: true, | ||
writable: true | ||
}); | ||
} | ||
size(chunk) { | ||
return chunk.byteLength; | ||
} | ||
} | ||
|
||
defineProperty(global, 'ByteLengthQueuingStrategy', { | ||
There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. These should likely emit an experimental process warning on first access. There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. this api is not experimental There was a problem hiding this comment. Choose a reason for hiding this commentThe reason will be displayed to describe this comment to others. Learn more. See above. I'd want to land this entire implementation as experimental first. |
||
value: ByteLengthQueuingStrategy, | ||
enumerable: false, | ||
configurable: true, | ||
writable: true | ||
}); | ||
}); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm definitely not convinced that these should be globals right out the gate. Similar to URL, TextDecoder, and TextEncoder, making them non-global at least while they are still experimental makes sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
only the added node stream apis are experimental. the whatwg stream implementations are not experimental.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
As we've done with the introduction of all WHATWG APIs, this entire implementation should land as experimental first until we're absolutely certain that it's definitely stable and has some use.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
the implementation is just copy-pasted from chromium, its almost 3 years old at this point, and has more tests than most stuff in node.js core. i see no reason to consider it experimental by that measurement. I have marked the APIs I wrote as experimental
i have some reservations around firing experimental warnings on access for them but if others also care i can add it in.