Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Bring in a faster Transform stream #147

Open
mcollina opened this issue Feb 8, 2024 · 6 comments
Open

Bring in a faster Transform stream #147

mcollina opened this issue Feb 8, 2024 · 6 comments

Comments

@mcollina
Copy link
Member

mcollina commented Feb 8, 2024

https://github.com/mcollina/syncthrough offers 3x the throughput of Transform. https://www.npmjs.com/package/minipass offer similar characteristics.

I think bringing in a faster set of Streams that enforce backpressure would speed up an area of Node.js that is showing its time.

@rluvaton
Copy link
Member

rluvaton commented Feb 9, 2024

I don't think we should have another stream implementation, this will be confusing and need to maintain multiple implementations for complex topic

@ronag
Copy link
Member

ronag commented Feb 9, 2024

There is room to further optimize stream.Transform to be close to synchthrough. To begin with we should avoid the double buffering and implement stream.Transform as a stream.Readable with stream.Writable compatible api.

@ronag
Copy link
Member

ronag commented Feb 9, 2024

Also rather than adding another stream api, I think we should add support for sync functions in pipeline, i.e.

stream.pipeline(src, val => val.toUpperCase(), res)

@mcollina
Copy link
Member Author

mcollina commented Feb 9, 2024

There is room to further optimize stream.Transform to be close to synchthrough. To begin with we should avoid the double buffering and implement stream.Transform as a stream.Readable with stream.Writable compatible API.

I would love this, whoever the world depends on _readableState and _writableState. If you have ideas, I'm all ears.

Also rather than adding another stream api, I think we should add support for sync functions in pipeline

@ronag that would have almost the same effect. Maybe we could internalize SyncThrough or analog and use ti to support pipeline

@rluvaton
Copy link
Member

@ronag I remember u had some pr to remove the double buffering but stopped, do you remember what blocked you?

@ronag
Copy link
Member

ronag commented Feb 11, 2024

@ronag I remember u had some pr to remove the double buffering but stopped, do you remember what blocked you?

Time was a blocker.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

3 participants