Add support for reading from a WebStreams

[Borewit]

Changes:

• Add support for reading from a WebStreams
• Stream Blob via a WebStreams, instead of buffering the full content

Moves remaining native Node Readable streams to the Node.js specific entry point (index.js). As the fileTypeStream is based on the the native Node Readable stream, that function is no longer available in the primary (other then Node engine) entry point (core.js).

⚠️ fromBlob() requires Node.js ≥ 20 as the strtok3.fromStream requires a Readable from which a ReadableStreamBYOBReader can be created. Which is only available since Node.js 20.

• Resolves Make Blob lazy #594
• Resolves Support reading from browser streams in server-side export #617
• Depends on: Web streams Borewit/strtok3#1098

Updating dependencies to:

graph TD;
    FTN("file-type (Node.js entry point)")-->FTP
    FTP("file-type (primary entry point)")-->S
    S(strtok3)-->P(peek-readable)
    S(strtok3)-->TO("@tokenizer/token")
    TY(token-types)-->TO
    TY-->IE("ieee754")
    FTP-->TY
    NS("node:stream")
    FTN-->NS
    FTP-->UAE(uint8array-extras)
    style NS fill:#F88,stroke:#A44
    style IE fill:#CCC,stroke:#888
    style FTN fill:#FAA,stroke:#A44

Loading

Previous dependency diagram can be found here: #578 (comment)

[sindresorhus]

I would prefer to have a single fromStream method that supports both.

[Borewit]

I would prefer to have a single fromStream method that supports both.

That was quick.
I will give that some thought, I like to release this one together #633, as joined effort to work without Node Streams and Buffer.

[Borewit]

I would prefer to have a single fromStream method that supports both.

@sindresorhus, are you not concerned that this make it more difficult to for tree-shaking to exclude node:buffer and nodes:stream when these are effectively not being used?

[sindresorhus]

I was thinking that fromStream in core.js would only support web streams, same with the one in browser.js, but in index.js it would get the additional support for Node.js streams. That way, only Node.js users gets the Node.js imports.

[sindresorhus]

Alternatively, we could just drop support for Node.js streams, and just tell users to call .toWeb() on their Node.js streams. That may be even better actually.

[sindresorhus]

Alternatively, we could just drop support for Node.js streams, and just tell users to call .toWeb() on their Node.js streams. That may be even better actually.

Let's go with this.

[Borewit]

Alternatively, we could just drop support for Node.js streams, and just tell users to call .toWeb() on their Node.js streams. That may be even better actually.

Let's go with this.

The [toWeb() method] (https://nodejs.org/api/stream.html#streamreadabletowebstreamreadable) is still experimental.

Maybe first release with as a hybrid solution, including Node.js & Web Streams. And indeed, moving the Node.js dependencies to the Node.js specific entry point index.js . Also note that in addition to the fileTypeFromStream(), there is the fileStream() which is still depends on Node.js streams, and also shared to with the primary entry point. I am honestly not very motivated to update that method, as it not complaint with the current architecture where file-type has access to the entire file. The argument for backward compatibility is now fading, as Web Streams are now becoming widely available.

[sindresorhus]

Maybe first release with as a hybrid solution, including Node.js & Web Streams. And indeed, moving the Node.js dependencies to the Node.js specific entry point index.js .

Ok, sure. But at least add a TODO comment to remove Node.js streams support and recommend toWeb() instead when it's stable.

[sindresorhus]

there is the fileStream() which is still depends on Node.js streams

It could be moved to index.js as it's already documented to only be available on Node.js.

[sindresorhus]

This type should be available globally, so I don't think we need to import it.

[Borewit]

The global type in incompatible with Node.js type. If I change it here, the problem will appear elsewhere,

[sindresorhus]

I'm not sure what that means, but this is core.d.ts, so it shouldn't import types only available for Node.js.

[Borewit]

The Node.js Readable types are incompatible with the global lib.dom.d.ts types, one things the challenges mentioned in #588 (comment), related to this issue DefinitelyTyped/DefinitelyTyped#60377, which unfortunately got closed with PR aiming to resolve that.

I used the Node.js, as Node.js has been the primary supported platform. Using the lib.dom.d.ts I need hack in types mapping cast somewhere.

Maybe good accept both, an do an ugly type cast, just for the convenience for our users.

Off-topic: The other mind f*ck, was the BYOB Readable Stream, The "Bring-Your-Own-Buffer" reader:

defines a reader for a ReadableStream that supports zero-copy reading from an underlying byte source

Well the first things the zero-copy method does is hijacking the buffer you bring, and essentially turns into junk (there is formal property for this state, forgot the name), meaning it can no longer be used. Which essentially forces you to create a new Buffer (as it becomes totally useless after providing it), and then copying the data to the Buffer you wanted to have it written in the first place. So the only feature BYOB has, versus the ReadableStreamDefaultReader is that you can control the chunk length to be written. How confusing, what a disappointment.

[sindresorhus]

Shouldn't this and toDetectionStream support web streams too?

[Borewit]

Yeah, should probably be done as well. (see #636)

[Borewit]

there is the fileStream() which is still depends on Node.js streams

It could be moved to index.js as it's already documented to only be available on Node.js.

That is very doable. We can raise an issue, to implement the same concept using Web Streams. I actually starting working on that, but it takes some time. Tricky stuff those streams.

Update, raised issue: #636 to implementing detecting web stream

[sindresorhus]

@Borewit Are we ready to do another release now or do you plan more changes? None of these changes are technically breaking, right? Since Buffer is also a Uint8Array, so 00e051b should not be breaking (it's only breaking if we would have changed a return value from Buffer to Uint8Array).

[Borewit]

@Borewit Are we ready to do another release now or do you plan more changes? None of these changes are technically breaking, right? Since Buffer is also a Uint8Array, so 00e051b should not be breaking (it's only breaking if we would have changed a return value from Buffer to Uint8Array).

This morning I realized, what could be documented better is the NodeFileTypeParser class

file-type/index.js

Line 9 in b815b5e

export class NodeFileTypeParser extends FileTypeParser {

which should be used instead of the FileTypeParser, for those implementing custom extensions and usign Node.js.

No objection to release now.

[Borewit]

Part of v19.1.0