Large build output cannot be uploaded

[asquithea]

I found that artifacts from the end of my build pipeline (just before deploy) are not cached.
Bit of a show-stopper as it causes a full redeploy!

It looks like Fastify has a default body size limit of 1 MiB, and so I'm speculating this is the reason.

Could we have a configuration option to bump this considerably higher?

A backport to the last stable version (so 1.1.2 I guess) would be really helpful, BTW.

[fox1t]

This is strange. We already have the limit increased from default here: https://github.com/fox1t/turborepo-remote-cache/blob/main/src/plugins/remote-cache/index.ts#L18

[asquithea]

Hmm. I'm building individually packaged lambda functions using esbuild, so the packages are varying between about 1 MB and 5 MB. But there are quite a lot of them in one service... the total size of the service build folder containing the ZIP files is 228 MB! This probably explains it if they're all being uploaded together as a single artifact: no reason to think your limit is ineffective.

100 MB does seem like a reasonable limit if the data is being buffered in memory. Do you think it would be possible to stream the request to storage? Along the lines of fastify/fastify#534 ? I appreciate this is probably harder to implement, though it might yield a performance benefit.

[fox1t]

I think we can support arbitrary significant artifacts using a strategy similar to one I used here: https://github.com/fox1t/fastify-multer (multer is only for form-data/multipart).

The idea is to have a content parser that flags the incoming request as "interesting" and then handles the upload in a preHanlder function, piping the incoming stream directly to a writable stream (regardless of the local file or remote bucket).

Would you like to open a PR to support it?

[tm1000]

This might also be related to vercel/turborepo#2096 and vercel/turborepo#2280

[tm1000]

After further testing and creating a PR in turborepo (vercel/turborepo#2428)

I can confirm that I was able to upload an artifact of 92mb to the ducktors/turborepo-remote-cache server implementation. It took about 1min

[simonjpartridge]

I have an issue where my artifacts are larger than the 104mb file size limit.

Would it be possible to increase the default, or add an environment variable to set this limit for the server?

[tm1000]

I'd like the same as @simonjpartridge. An environment variable would be great