(fix gatsby - redux cache) Reduced chunk size buffer from 150 gb to 1.5 gb

[bebrian458]

Description

One of my team members @j218 noticed a buffer issue when trying to run build on our gatsby app with large amounts of rich text content.

We were able to narrow it down to a redux caching issue. There seems to have been an attempt in fixing this by chunking the nodes. However, we noticed a slight math error. The intention was to chunk nodes to fit into a 1.5gb buffer, but the calculations indicate chunking for a 150gb buffer. Assuming that the max buffer length of node is 2GB, this calculation issue would create a huge chunk size if the the entries contained a large amount of content. Thus, the issue would always cause the number of chunks to equal one (essentially performing no actual chunking) and cause the issue shown in the screenshot above.

We have made the fix to chunk the nodes to fit into a 1.5 gb cache.

Fixes #17233

[blainekasten]

cc @pvdz

[blainekasten]

@bebrian458 a sort of weird nit, do you mind getting rid of the commits that include Hashim's 100 days of Gatsby blog post?

[pvdz]

That's not a weird nit! My typo isn't weird! Code review isn't weird!

But also, thank you. That's a weird error indeed :(

[pvdz]

But. That probably means my fixes unblocked you. I'm happy about that :D (If it weren't for my fix, you would be hitting that Buffer length limit not due to a bug, but simply because it wasn't designed to support that size :) )

Can you share some stats about your site? Why are you hitting this limit? Large page count or large page contents? How long does it take to build? What kind of site is it? Cu-ri-ous!

[gatsbot]

Holy buckets, @bebrian458 — we just merged your PR to Gatsby! 💪💜

Gatsby is built by awesome people like you. Let us say “thanks” in two ways:

1. We’d like to send you some Gatsby swag. As a token of our appreciation, you can go to the Gatsby Swag Store and log in with your GitHub account to get a coupon code good for one free piece of swag. We’ve got Gatsby t-shirts, stickers, hats, scrunchies, and much more. (You can also unlock even more free swag with 5 contributions — wink wink nudge nudge.) See gatsby.dev/swag for details.
2. We just invited you to join the Gatsby organization on GitHub. This will add you to our team of maintainers. Accept the invite by visiting https://github.com/orgs/gatsbyjs/invitation. By joining the team, you’ll be able to label issues, review pull requests, and merge approved pull requests.

If there’s anything we can do to help, please don’t hesitate to reach out to us: tweet at @gatsbyjs and we’ll come a-runnin’.

Thanks again!

[blainekasten]

Just published a patch under gatsby@2.19.29. Thanks for the quick PR!