Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Prevent Axios from fully buffering files uploaded from MSP dashboard #24927

Merged
merged 1 commit into from
Dec 19, 2024

Conversation

sgress454
Copy link
Contributor

for #24829

See axios/axios#1045 -- by default Axios buffers uploaded files into memory fully, to support redirects. For large file uploads this means we get OOM errors, especially when sending to multiple teams. There's a few other optimizations we can put in place here but in the short term we can fix the buffering issue by setting maxRedirects: 0 on the requests.

I tested this by adding an onUploadProgress handler to the Axios request that dumps memory usage, and uploading a 209mb software file to 3 teams. Before the update, the readout ticket up continuously (the first number is the # of bytes uploaded):

1540129 {rss: 161652736, heapTotal: 65880064, heapUsed: 55625552, external: 28411157, arrayBuffers: 24338844}
edit-software.js:177
1554313 {rss: 149254144, heapTotal: 65880064, heapUsed: 52445200, external: 25193635, arrayBuffers: 21121327}
edit-software.js:177
2339833 {rss: 151703552, heapTotal: 66404352, heapUsed: 52269280, external: 12664377, arrayBuffers: 8592064}
...a minute later...
192708641 {rss: 619323392, heapTotal: 95240192, heapUsed: 55320960, external: 618952429, arrayBuffers: 614879965}
edit-software.js:177
201523233 {rss: 634613760, heapTotal: 95240192, heapUsed: 58514992, external: 636581613, arrayBuffers: 632509154}
edit-software.js:177
209326677 {rss: 637399040, heapTotal: 95240192, heapUsed: 56800016, external: 639441633, arrayBuffers: 635369173}

so we start at ~161mb, and by the time we're done, we're using 637mb of RAM. Render's free tier has a 250mb limit on apps.

With maxRedirects: 0, we see:

2669337 {rss: 151846912, heapTotal: 66404352, heapUsed: 53297400, external: 26446868, arrayBuffers: 22374419}
edit-software.js:177
2279929 {rss: 152641536, heapTotal: 66404352, heapUsed: 53453664, external: 27233300, arrayBuffers: 23160851}
edit-software.js:177
2228585 {rss: 153038848, heapTotal: 66404352, heapUsed: 53537096, external: 27626516, arrayBuffers: 23554067}
...a minute later...
209326677 {rss: 146989056, heapTotal: 92094464, heapUsed: 53802856, external: 14617518, arrayBuffers: 10545071}
edit-software.js:177
209326677 {rss: 153051136, heapTotal: 92094464, heapUsed: 55376336, external: 22447478, arrayBuffers: 18375026}
edit-software.js:177
209326677 {rss: 152129536, heapTotal: 92094464, heapUsed: 51857632, external: 22447478, arrayBuffers: 16540013}

showing that we start and finish with around the same amount of RAM used.

@sgress454 sgress454 merged commit 556b79e into main Dec 19, 2024
7 checks passed
@sgress454 sgress454 deleted the sgress454/24829-msp-multi-upload branch December 19, 2024 21:51
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Development

Successfully merging this pull request may close these issues.

2 participants