simplify streamToString method from node-web-streams.helper.ts
#62841
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Ethan-Arrowood
requested review from
timneutkens,
ijjk,
shuding,
huozhi,
feedthejim,
ztanner and
wyattjoh
as code owners
Stats from current PRDefault BuildGeneral
Client Bundles (main, webpack)
Legacy Client Bundles (polyfills)
Client Pages
Client Build Manifests
Rendered Page Sizes
Edge SSR bundle Size
Middleware size
Next Runtimes
build cache
Diff detailsDiff for middleware.jsDiff too large to display Diff for edge-ssr.jsDiff too large to display Diff for app-page-exp..ntime.dev.jsDiff too large to display Diff for app-page-exp..time.prod.jsDiff too large to display Diff for app-page-tur..time.prod.jsDiff too large to display Diff for app-page-tur..time.prod.jsDiff too large to display Diff for app-page.runtime.dev.jsfailed to diffDiff for app-page.runtime.prod.jsDiff too large to display Diff for pages-turbo...time.prod.jsDiff too large to display Diff for pages.runtime.dev.jsDiff too large to display Diff for pages.runtime.prod.jsDiff too large to display Diff for server.runtime.prod.jsDiff too large to display |
|
I'm seeing about a 50% increase 😄
const assert = require('assert')
const fs = require('fs');
const stream = require('stream');
const path = require('path');
function createDecodeTransformStream(decoder = new TextDecoder()) {
return new TransformStream({
transform(chunk, controller) {
return controller.enqueue(decoder.decode(chunk, { stream: true }))
},
flush(controller) {
return controller.enqueue(decoder.decode())
},
})
}
async function oldStreamToString (stream) {
let buffer = ''
await stream
// Decode the streamed chunks to turn them into strings.
.pipeThrough(createDecodeTransformStream())
.pipeTo(
new WritableStream({
write(chunk) {
buffer += chunk
},
})
)
return buffer
}
async function newStreamToString (stream) {
const decoder = new TextDecoder()
let string = ''
for await (const chunk of stream) {
string += decoder.decode(chunk, { stream: true })
}
string += decoder.decode()
return string
}
async function benchOld (p, exp) {
const r = fs.createReadStream(p);
const rs = stream.Readable.toWeb(r);
const s = performance.now();
const act = await oldStreamToString(rs);
const e = performance.now();
assert.strictEqual(act, exp);
return e - s;
}
async function benchNew (p, exp) {
const r = fs.createReadStream(p);
const rs = stream.Readable.toWeb(r);
const s = performance.now();
const act = await newStreamToString(rs);
const e = performance.now();
assert.strictEqual(act, exp);
return e - s;
}
function avg (l) {
let t = 0;
for (const i of l) {
t += i;
}
return t / l.length;
}
async function bench () {
const p = path.join(__dirname, './input.txt');
const exp = fs.readFileSync(p, 'utf-8');
// warmup;
await benchOld(p, exp);
await benchNew(p, exp);
const results = [[], []]; // old, new
for (let i = 0; i < 5; i++) {
const o = await benchOld(p, exp);
const n = await benchNew(p, exp);
results[0].push(o);
results[1].push(n);
}
console.log(results);
console.log('avg time of old = ', avg(results[0]))
console.log('avg time of new = ', avg(results[1]))
}
bench(); |
Failing test suitesCommit: d483134
Expand output● app dir - with output export - dynamic api route prod › production mode › should work in prod with dynamicPage 'force-static' Read more about building and testing Next.js in contributing.md. |
feedthejim
reviewed
Mar 5, 2024
| string += decoder.decode(chunk, { stream: true }) | ||
| } | ||
|
|
||
| string += decoder.decode() |
There was a problem hiding this comment.
Why is a last decode call needed?
There was a problem hiding this comment.
https://nodejs.org/api/util.html#textdecoderdecodeinput-options
Decodes the input and returns a string. If options.stream is true, any incomplete byte sequences occurring at the end of the input are buffered internally and emitted after the next call to textDecoder.decode().
There was a problem hiding this comment.
Essentially, this makes sure everything is decoded, even if there are incomplete bytes.
gnoff
reviewed
Mar 5, 2024
packages/next/src/server/stream-utils/node-web-streams-helper.ts
Outdated
Show resolved
Hide resolved
gnoff
approved these changes
Mar 5, 2024
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
The
encode-decode.tsfile is completely replaceable with theTextDecoderStreamapi, but also we can simplify thestreamToStringfunction too.Working on some benchmarks now - wanted to get CI running to see if this breaks anything though.