Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Increase exec max buffer size to support large amounts of files #18

Merged
merged 3 commits into from
Aug 15, 2023

Conversation

jieey1140
Copy link
Contributor

@jieey1140 jieey1140 commented Aug 15, 2023

I'm going to build the nestjs app as isolate and use it in firebase deploy.

The following error occurs when executing isolation after nest build.

error RangeError: stderr maxBuffer length exceeded
    at new NodeError (node:internal/errors:372:5)
    at Socket.onChildStderr (node:child_process:485:14)
    at Socket.emit (node:events:527:28)
    at addChunk (node:internal/streams/readable:315:12)
    at readableAddChunk (node:internal/streams/readable:285:11)
    at Socket.Readable.push (node:internal/streams/readable:228:10)
    at Pipe.onStreamRead (node:internal/stream_base_commons:190:23) 
 //   Below is the log I captured while debugging:
Error: Error: Error: RangeError [ERR_CHILD_PROCESS_STDIO_MAXBUFFER]: stderr maxBuffer length exceeded
    at pack (file:///Users/gimsehun/Documents/PROJECT-main/node_modules/.pnpm/isolate-package@1.3.1/node_modules/isolate-package/src/utils/yaml.ts:15:7)
    at processTicksAndRejections (node:internal/process/task_queues:96:5)
    at processBuildOutputFiles (file:///Users/gimsehun/Documents/PROJECT-main/node_modules/.pnpm/isolate-package@1.3.1/node_modules/isolate-package/src/index.ts:76:3)
    at start (file:///Users/gimsehun/Documents/PROJECT-main/node_modules/.pnpm/isolate-package@1.3.1/node_modules/isolate-package/dist/index.mjs:768:3),[object Promise]
    at process.error (file:///Users/gimsehun/Documents/PROJECT-main/node_modules/.pnpm/isolate-package@1.3.1/node_modules/isolate-package/src/utils/pack.ts:3:8)
    at process.emit (node:events:527:28)
    at process.emit (/Users/gimsehun/Documents/PROJECT-main/node_modules/.pnpm/source-map-support@0.5.21/node_modules/source-map-support/source-map-support.js:516:21)
    at emit (node:internal/process/promises:140:20)
    at processPromiseRejections (node:internal/process/promises:274:27)
    at processTicksAndRejections (node:internal/process/task_queues:97:32)

We have identified that the excessive number of src files is causing issues, so we have increased the maxBuffer size in exec to accommodate repositories with a large number of files. (200KB(Default) -> 10MB)

After making this change and conducting tests, we have confirmed that the (JavaScript compiled) environment now compiles successfully!

Copy link
Owner

@0x80 0x80 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for this PR 👍 One small change..

I will also change the title of the PR so it can be merged as a descriptive commit message.

src/utils/pack.ts Outdated Show resolved Hide resolved
@0x80 0x80 changed the title maxBuffer error occurs during isolate build. Increase exec max buffer size to support large amounts of files Aug 15, 2023
@jieey1140 jieey1140 requested a review from 0x80 August 15, 2023 14:42
@jieey1140
Copy link
Contributor Author

@0x80 Thank you for the quick verification!

I've reviewed the comment, and I agree that "options" seems better than "option." It has been updated :)

@jieey1140
Copy link
Contributor Author

@0x80 Additional note: Thanks to the package you've created, we've been able to develop numerous common packages within our monorepo, resulting in a significant reduction of duplicated code. We wanted to take this opportunity to express our heartfelt gratitude! 👍

@0x80
Copy link
Owner

0x80 commented Aug 15, 2023

@jieey1140 I'm happy to hear that people find it useful.

My next step is to get it integrated into the official firebase-tools deploy command, as it allows for running the emulator still with changes hot reloading, but I'm a little swamped with other work so I don't know when I'll give that a try :)

@0x80 0x80 merged commit a972a0e into 0x80:main Aug 15, 2023
1 check passed
@0x80
Copy link
Owner

0x80 commented Aug 15, 2023

I have released version 1.3.3 with this change

@jieey1140
Copy link
Contributor Author

@0x80 Thank you. I've verified that the build was successful without any errors. Have a great day!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants