stream: allow typed arrays to be written and read #22427

SirR4T · 2018-08-21T08:21:47Z

Checklist

make -j4 test (UNIX), or vcbuild test (Windows) passes
tests and/or benchmarks are included
documentation is changed or added
commit message follows commit guidelines

Partially fixing #1826

Caveat

Any TypedArrays can be written, now, but in non-objectMode, the .write() method still receives raw Buffers.

I'm not completely sure that is the behaviour we want, but I'm willing to work towards fixing it, with some mentoring.

SirR4T · 2018-08-21T08:39:16Z

on second thought, the Uint8Array support for streams does the same thing as well, so 🤷‍♂️

mscdex · 2018-08-21T12:43:51Z

lib/stream.js

+      '[object Float32Array]',
+      '[object Float64Array]',
+      '[object DataView]'
+    ].includes(Object.prototype.toString.call(obj));


This is going to be pretty slow. Surely there is a better way...

Will .indexOf() be faster? Open to other suggestions as well.

Can't we just use something like ArrayBuffer.isView()? It seems to be available since at least node v4. The reason the string check was used before was because we were explicitly checking for only Uint8Array.

On util.types are a lot of checks that can be used as well.

@BridgeAR the relevant function from util.types is already being used earlier in this file if it exists. This implementation here is used if both the function from util.types and the function from the util binding are not available.

@mscdex so ArrayBuffer.isView() could still be available, even though util.types and util binding are not available?
In other PRs dealing with typed arrays, i have been using:

const { isArrayBufferView } = require('internal/util/types');

Same can be used here too?
Wasn't sure whether compatibility with 0.x versions was specifically required here, and how to add that support.

so ArrayBuffer.isView() could still be available, even though util.types and util binding are not available?

My guess is that a string-based check was used previously for better compatibility with other, non-node platforms that use the same streams interface? I do not know for sure. Perhaps @addaleax might remember why they added that kind of implementation?

EDIT: judging by the comments from earlier in this file, it could also be useful for node v6.x, which does not have require('util').types, process.binding('util').isArrayBufferView, or require('internal/util/types').

So i'm guessing current PR stays put?

Or is there any merit in adding a patch of this sort:

diff --git a/lib/stream.js b/lib/stream.js index c6861b3464..58aab7c16d 100644 --- a/lib/stream.js +++ b/lib/stream.js @@ -42,6 +42,15 @@ Stream.finished = eos; Stream.Stream = Stream; // Internal utilities +try { + const { isArrayBufferView } = require('internal/util/types'); + if (typeof isArrayBufferView === 'function') { + Stream._isArrayBufferView = isArrayBufferView; + } +} catch (e) { +} + +if (!Stream._isArrayBufferView) { try { const types = require('util').types; if (types && typeof types.isArrayBufferView === 'function') { @@ -53,6 +62,7 @@ try { } } catch (e) { } +} if (!Stream._isArrayBufferView) { Stream._isArrayBufferView = function _isArrayBufferView(obj) {

@SirR4T That's more or less the exact same thing that's being done below that with require('util').types. I still think using ArrayBuffer.isView() as the fallback is probably best. I suppose the function could be cached at startup like internal/util/types does.

Updated to use ArrayBuffer.isView. Although do verify, if simply having it in a constant will cache it? Or should the const isTypedArray be moved to module scope?

Also, shouldn't we be checking for existence of ArrayBuffer.isView first, and then be falling back to other methods (util.types)?

mscdex · 2018-08-21T17:36:22Z

lib/stream.js

@@ -45,37 +45,50 @@ Stream.Stream = Stream;
 try {
  const types = require('util').types;
  if (types && typeof types.isUint8Array === 'function') {
-    Stream._isUint8Array = types.isUint8Array;
+    // Stream._isUint8Array = types.isUint8Array;


These comments should be removed

BridgeAR · 2018-08-21T17:47:47Z

lib/stream.js

+      '[object Float32Array]',
+      '[object Float64Array]',
+      '[object DataView]'
+    ].includes(Object.prototype.toString.call(obj));


On util.types are a lot of checks that can be used as well.

BridgeAR · 2018-08-21T17:49:30Z

lib/stream.js

-  if (!Stream._uint8ArrayToBuffer) {
-    Stream._uint8ArrayToBuffer = function _uint8ArrayToBuffer(chunk) {
+  if (!Stream._typedArrayToBuffer) {
+    Stream._typedArrayToBuffer = function _typedArrayToBuffer(chunk) {
      return Buffer.prototype.slice.call(chunk);


This will likely not work with anything besides Uint8Array. So any other TypedArray would fail.

updated ._typedArrayToBuffer to use Buffer.from().

mscdex · 2018-08-21T18:03:41Z

lib/stream.js

@@ -45,37 +45,50 @@ Stream.Stream = Stream;
 try {
  const types = require('util').types;
  if (types && typeof types.isUint8Array === 'function') {


This needs to be updated to check for the correct function.

SirR4T

Hi @BridgeAR , does this look ok, for _typedArrayToBuffer()?

SirR4T · 2018-08-22T15:18:21Z

lib/stream.js

@@ -87,7 +87,7 @@ if (version[0] === 0 && version[1] < 12) {

  if (!Stream._typedArrayToBuffer) {
    Stream._typedArrayToBuffer = function _typedArrayToBuffer(chunk) {
-      return Buffer.prototype.slice.call(chunk);
+      return Buffer.from(chunk.buffer, chunk.byteOffset, chunk.byteLength);


the Buffer.prototype.slice() would have made a different Buffer. With Buffer.from(), the underlying memory will still be shared. Will that be a problem?

mscdex · 2018-08-23T09:33:12Z

lib/stream.js

+if (!Stream._isArrayBufferView) {
+  // Cached to make sure no userland code can tamper with it.
+  const isArrayBufferView = ArrayBuffer.isView;
+  Stream._isArrayBufferView = isArrayBufferView;


Oh, in this case we don't actually need to do any caching. I think we can just do

Stream._isArrayBufferView = ArrayBuffer.isView;

SirR4T · 2018-08-27T06:38:09Z

@mscdex / @BridgeAR : any other changes suggested / required here?

mscdex · 2018-08-27T08:39:13Z

CI: https://ci.nodejs.org/job/node-test-pull-request/16777/

SirR4T · 2018-08-29T11:19:34Z

rebased against master. does that warrant a new CI?

SirR4T · 2018-08-30T05:04:05Z

Hi @mscdex , sorry for the commit and CI noise, but I had rebased this branch against master, after the earlier CI run had completed.

For this to be merged in, should another CI run be initiated?

Also, kind of off topic:
I personally prefer to keep my feature branches updated with (rebased against) master, so that builds across different feature branches may benefit simultaneously. Is this practice frowned upon? Or are there other ways to achieve the same?

mcollina · 2018-09-03T07:43:57Z

I've added the dont-land-on-6 and dont-land-on-8 labels out of caution.

mcollina · 2018-09-03T07:24:12Z

lib/stream.js

-  if (types && typeof types.isUint8Array === 'function') {
-    Stream._isUint8Array = types.isUint8Array;
+  if (types && typeof types.isArrayBufferView === 'function') {
+    Stream._isArrayBufferView = types.isArrayBufferView;
  } else {
    // This throws for Node < 4.2.0 because there's no util binding and
    // returns undefined for Node < 7.4.0.


This comment would need to be updated. When has isArrayBufferView been added?

mcollina · 2018-09-03T07:30:14Z

doc/api/stream.md

+* `chunk` {Buffer|TypedArray|DataView|string|null|any} Chunk of data to push
+  into the read queue. For streams not operating in object mode, `chunk` must
+  be a string, `Buffer` or `Uint8Array`. For object mode streams, `chunk` may
+  be any JavaScript value.


I guess this sentence is not up-to-date. It's not just string, Buffer or Uint8Array anymore.

True, but the typed arrays and data views can be ~~passed~~ received only in object mode. So thought that the current statement still remains true.

It might be better to explicitly document this behaviour though. Any thoughts on the same?

Maybe I misunderstood this PR then. I thought the whole point of this is to support all TypedArray and DataView in binary mode.

This looks like from looking at the code as !state.objectMode && Stream._isArrayBufferView works when objectMode is `false.

yea... I wasn't sure of the behaviour we required here, which is why I had raised the PR with a caveat.

The PR introducing support for Uint8Array in streams, too, asserts a similar behaviour, so I had let this PR follow similarly.

Exactly. Still, this PR make accepting TypedArray and DataView everywhere so it should be reflected in the docs.

mcollina · 2018-09-03T07:43:18Z

lib/stream.js

-      return Buffer.prototype.slice.call(chunk);
+  if (!Stream._typedArrayToBuffer) {
+    Stream._typedArrayToBuffer = function _typedArrayToBuffer(chunk) {
+      return Buffer.from(chunk.buffer, chunk.byteOffset, chunk.byteLength);
    };
  }
 }


I think we can clean up this whole file. As long as it exports the two functions we need, it's ok to not have all those checks. readable-stream@3 supports Node 6+ anyway.

mcollina · 2018-09-03T07:45:55Z

CI: https://ci.nodejs.org/job/node-test-pull-request/16982/

addaleax · 2018-09-03T12:51:50Z

Resume CI: https://ci.nodejs.org/job/node-test-pull-request/16988/

mcollina · 2018-09-04T07:56:10Z

Resume CI: https://ci.nodejs.org/job/node-test-pull-request/17003/
CITGM (master): https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1527/
CITGM: https://ci.nodejs.org/view/Node.js-citgm/job/citgm-smoker/1528

mcollina · 2018-09-04T08:43:26Z

doc/api/stream.md

+* `chunk` {Buffer|TypedArray|DataView|string|null|any} Chunk of data to push
+  into the read queue. For streams not operating in object mode, `chunk` must
+  be a string, `Buffer` or `Uint8Array`. For object mode streams, `chunk` may
+  be any JavaScript value.


Exactly. Still, this PR make accepting TypedArray and DataView everywhere so it should be reflected in the docs.

test/parallel/test-stream-typedarrays.js

SirR4T · 2018-09-04T10:59:50Z

removed legacy compatibility code for lib/stream.js (cc: @mcollina , @mscdex , and @addaleax due to comments here and here)
added tests for streams where writeable is piped to readable, and with transforms,
updated docs for typed arrays and data views,
added docs explicitly stating conversion of typed arrays and data views in non-object mode.
rebased against current master,

mcollina · 2018-09-04T12:23:57Z

test/parallel/test-stream-transform-objectmode-typedarrays.js

+// NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM,
+// DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR
+// OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE
+// USE OR OTHER DEALINGS IN THE SOFTWARE.


this header is not required

mcollina · 2018-09-04T12:26:07Z

test/parallel/test-stream-transform-objectmode-typedarrays.js

+
+const src = new PassThrough({ objectMode: true });
+const tx = new PassThrough({ objectMode: true });
+const dest = new PassThrough({ objectMode: true });


the whole point of this change is see how they work with objectMode: false.

mcollina · 2018-09-04T12:26:29Z

test/parallel/test-stream-pipe-typedarrays-flow.js

+  let ticks = testBuffers.length;
+
+  const rs = new Readable({
+    objectMode: true,


this needs to be objectMode: false. Also everywhere else.

SirR4T · 2018-09-05T04:10:29Z

@mcollina : pushed changes with objectMode: false, and some changes to internal _stream_readable.js and _stream_writable.js to remove checks which I think are unnecessary now. They might have been necessary before Buffer was a subclass of Uint8Array.

One test is still failing, which raises this question: how should encoding work, for any .write() methods where chunk is not a string? As far as I understand, it works with Buffers and Uint8Arrays currently (before this PR), but will not work with any other typed array or data view.

mcollina · 2018-09-05T10:49:31Z

pushed changes with objectMode: false, and some changes to internal _stream_readable.js and _stream_writable.js to remove checks which I think are unnecessary now. They might have been necessary before Buffer was a subclass of Uint8Array.

Are you sure? Can you please add some checks in the tests that everything that is emitted/returned by on('data') and read() are Buffers? Also, can you check that everything that gets into _write() and _writev() are indeed Buffer?

One test is still failing, which raises this question: how should encoding work, for any .write() methods where chunk is not a string? As far as I understand, it works with Buffers and Uint8Arrays currently (before this PR), but will not work with any other typed array or data view.

I think this is why you need the conversion that you have commented out.

lundibundi · 2018-10-03T08:28:23Z

ping @SirR4T.

SirR4T · 2018-10-07T06:55:07Z

Currently swamped at work, will look into this whenever I find some time. Most likely within the next two weeks. Sarat.

…

On Wed, Oct 3, 2018 at 2:01 PM Denys Otrishko ***@***.***> wrote: ping @SirR4T <https://github.com/SirR4T>. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#22427 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AB8IMLS1rZyf_9rPyjgl5jI0WFR1C_jbks5uhHXVgaJpZM4WFUJ6> .

antsmartian · 2018-11-12T14:37:12Z

@SirR4T Ping, any update on this?

SirR4T · 2018-11-17T07:19:06Z

fixed the merge conflicts for now, but if someone could guide me on correcting the test cases, would appreciate the help.

oyyd · 2018-11-22T04:42:24Z

test/parallel/test-stream-typedarrays.js

+    write: common.mustCall((chunk, encoding, cb) => {
+      assert(!(chunk instanceof Buffer));
+      assert(ArrayBuffer.isView(chunk));
+      assert.deepStrictEqual(chunk, nthWrittenBuffer(n++));


Instances of Buffer could not be deepStrictEqual to those of TypedArray, could they?

Maybe try:

assert.deepStrictEqual(Buffer.from(chunk.buffer, chunk.byteOffset, chunk.length), nthWrittenBuffer(n++));

BridgeAR · 2019-03-10T00:44:28Z

@SirR4T what is the status here?

jasnell · 2020-07-07T14:36:33Z

Given that this is out of date and there has been no activity in over a year, closing. Can reopen if it is picked back up again

nodejs-github-bot added the stream Issues and PRs related to the stream subsystem. label Aug 21, 2018

SirR4T changed the title ~~Fix1826 stream typed arrays~~ stream: allow typed arrays to be written and read Aug 21, 2018

SirR4T mentioned this pull request Aug 21, 2018

net: allow typed arrays in .write() #22428

Closed

4 tasks

mscdex reviewed Aug 21, 2018

View reviewed changes

BridgeAR reviewed Aug 21, 2018

View reviewed changes

mscdex reviewed Aug 21, 2018

View reviewed changes

SirR4T commented Aug 22, 2018

View reviewed changes

mscdex reviewed Aug 23, 2018

View reviewed changes

TimothyGu mentioned this pull request Aug 28, 2018

Accept ArrayBuffer (and typed array/data view?) anywhere Buffer is allowed in the API #1826

Closed

SirR4T force-pushed the fix1826-stream-TypedArrays branch from fdcd563 to 38fd0fc Compare August 29, 2018 10:28

mcollina added semver-minor PRs that contain new features and should be released in the next minor version. dont-land-on-v6.x labels Sep 3, 2018

mcollina reviewed Sep 3, 2018

View reviewed changes

mcollina reviewed Sep 4, 2018

View reviewed changes

SirR4T force-pushed the fix1826-stream-TypedArrays branch from 38fd0fc to 865f456 Compare September 4, 2018 10:52

mcollina reviewed Sep 4, 2018

View reviewed changes

lundibundi added the wip Issues and PRs that are still a work in progress. label Oct 8, 2018

SirR4T force-pushed the fix1826-stream-TypedArrays branch from 06b494a to dbfcade Compare October 17, 2018 09:26

SirR4T and others added 11 commits November 17, 2018 13:05

stream: allow typed arrays

e8965c1

stream: add tests and docs for typedarrays

b758077

doc: update pr-urls for stream.md

883d957

stream: remove comments, and update function to check

91e092a

stream: fix Stream._typedArrayToBuffer

0930336

stream: fall back to ArrayBuffer.isView

10c57bc

stream: fix Stream._isArrayBufferView

51f2b9f

stream: remove legacy compatibility checks for internal functions

3e4668a

test: add transform and pipe tests for typed arrays

dca1cab

doc: update stream.md to for typed arrays

58ed18f

stream: fix reading and writing of typed arrays

7841da2

SirR4T force-pushed the fix1826-stream-TypedArrays branch from 9f31601 to 7841da2 Compare November 17, 2018 07:37

oyyd reviewed Nov 22, 2018

View reviewed changes

BridgeAR removed the dont-land-on-v6.x label Apr 26, 2019

Trott force-pushed the master branch from 1ecc406 to 49cf67e Compare September 17, 2019 16:52

BridgeAR removed the dont-land-on-v8.x label Jan 2, 2020

BridgeAR force-pushed the master branch 2 times, most recently from 8ae28ff to 2935f72 Compare May 31, 2020 12:20

jasnell closed this Jul 7, 2020

stream: allow typed arrays to be written and read #22427

stream: allow typed arrays to be written and read #22427

Conversation

SirR4T commented Aug 21, 2018

Checklist

Caveat

SirR4T commented Aug 21, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mscdex Aug 21, 2018 • edited Loading

Choose a reason for hiding this comment

SirR4T Aug 22, 2018 • edited Loading

Choose a reason for hiding this comment

mscdex Aug 22, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirR4T left a comment • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mscdex Aug 23, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirR4T commented Aug 27, 2018

mscdex commented Aug 27, 2018

SirR4T commented Aug 29, 2018

SirR4T commented Aug 30, 2018

mcollina commented Sep 3, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirR4T Sep 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirR4T Sep 4, 2018 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

mcollina commented Sep 3, 2018

addaleax commented Sep 3, 2018

mcollina commented Sep 4, 2018

Choose a reason for hiding this comment

SirR4T commented Sep 4, 2018

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

SirR4T commented Sep 5, 2018 • edited Loading

mcollina commented Sep 5, 2018

lundibundi commented Oct 3, 2018

SirR4T commented Oct 7, 2018 via email

antsmartian commented Nov 12, 2018

SirR4T commented Nov 17, 2018

Choose a reason for hiding this comment

BridgeAR commented Mar 10, 2019

jasnell commented Jul 7, 2020

mscdex Aug 21, 2018 •

edited

Loading

SirR4T Aug 22, 2018 •

edited

Loading

mscdex Aug 22, 2018 •

edited

Loading

SirR4T left a comment •

edited

Loading

mscdex Aug 23, 2018 •

edited

Loading

SirR4T Sep 4, 2018 •

edited

Loading

SirR4T Sep 4, 2018 •

edited

Loading

SirR4T commented Sep 5, 2018 •

edited

Loading