Gatsby build fails with Javascript heap out of memory when upgrading beyond 2.21.40

[tim-line]

Description

My build runs fine running Gatsby 2.21.40 but upgrading Gatsby beyond this version throws a javascript heap out of memory error during the build and develop process. Increasing max-old-space-size doesn't seem to make any difference (unless I am doing this incorrectly - I'm appending the following for the develop command when run from the command line: "node --max-old-space-size=8192 node_modules/gatsby/dist/bin/gatsby.js develop").

This appears to be currently failing at the 'createPages' step. The stack trace reads as:

<--- Last few GCs --->

[14608:000001B184536600]   368654 ms: Mark-sweep 2025.6 (2070.5) -> 2012.7 (2071.7) MB, 1356.9 / 0.1 ms  (average mu = 0.147, current mu = 0.095) allocation failure scavenge might not succeed

<--- JS stacktrace --->

==== JS stack trace =========================================

    0: ExitFrame [pc: 00007FF609BAA09D]
    1: StubFrame [pc: 00007FF609B2E803]
Security context: 0x03cdfd3808d1 <JSObject>
    2: resolve [000001A706AC26D9] [path.js:~130] [pc=000001B084A6E62B](this=0x02b1abaffd99 <Object map = 00000226453B0369>)
    3: arguments adaptor frame: 2->0
    4: readPackage(aka readPackage) [000001F1101994A1] [internal/modules/cjs/loader.js:~245] [pc=000001B084A19419](this=0x00a8503804b1 <undefined>,0x03bb8815...

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 00007FF608F9538F napi_wrap+114751
 2: 00007FF608F3FE26 v8::base::CPU::has_sse+66646
 3: 00007FF608F40C26 v8::base::CPU::has_sse+70230
 4: 00007FF609755B3E v8::Isolate::ReportExternalAllocationLimitReached+94
 5: 00007FF60973DC11 v8::SharedArrayBuffer::Externalize+833
 6: 00007FF609609E7C v8::internal::Heap::EphemeronKeyWriteBarrierFromCode+1436
 7: 00007FF6096150B0 v8::internal::Heap::ProtectUnprotectedMemoryChunks+1312
 8: 00007FF609611BD4 v8::internal::Heap::PageFlagsAreConsistent+3204
 9: 00007FF6096073D3 v8::internal::Heap::CollectGarbage+1283
10: 00007FF609605A44 v8::internal::Heap::AddRetainedMap+2452
11: 00007FF609626D8D v8::internal::Factory::NewFillerObject+61
12: 00007FF60938CED1 v8::internal::interpreter::JumpTableTargetOffsets::iterator::operator=+1665
13: 00007FF609BAA09D v8::internal::SetupIsolateDelegate::SetupHeap+546637
14: 00007FF609B2E803 v8::internal::SetupIsolateDelegate::SetupHeap+40627
15: 000001B084A6E62B

Steps to reproduce

Run the develop command using: 'node --max-old-space-size=8192 node_modules/gatsby/dist/bin/gatsby.js develop' or 'gatsby develop'

This is currently for the client project which is in development and uses a custom source plugin so I can't share the plugin publically. I may be able to share a test case privately if that is an option.

Expected result

Build should run without issue - it does prior to upgrading beyond 2.21.40

Actual result

Build fails with javascript heap out of memory error.

Environment

Gatsby info for failing build:

System:
OS: Windows 10 10.0.18362
CPU: (4) x64 Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
Binaries:
Node: 12.18.2 - C:\Program Files\nodejs\node.EXE
npm: 6.14.5 - C:\Program Files\nodejs\npm.CMD
Browsers:
Edge: 44.18362.449.0
npmPackages:
gatsby: ^2.24.4 => 2.24.4
gatsby-background-image: ^1.1.1 => 1.1.1
gatsby-image: ^2.4.13 => 2.4.13
gatsby-plugin-google-tagmanager: ^2.3.11 => 2.3.11
gatsby-plugin-instagram-embed: ^2.0.1 => 2.0.1
gatsby-plugin-manifest: ^2.4.18 => 2.4.18
gatsby-plugin-offline: ^3.2.18 => 3.2.18
gatsby-plugin-preact: ^3.3.2 => 3.3.2
gatsby-plugin-react-helmet: ^3.3.10 => 3.3.10
gatsby-plugin-remove-console: 0.0.2 => 0.0.2
gatsby-plugin-sharp: ^2.6.19 => 2.6.19
gatsby-plugin-twitter: ^2.3.10 => 2.3.10
gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4
gatsby-source-filesystem: ^2.3.20 => 2.3.20
gatsby-transformer-sharp: ^2.5.11 => 2.5.11
npmGlobalPackages:
gatsby-cli: 2.12.51

Gatsby info for succesful build:

System:
OS: Windows 10 10.0.18362
CPU: (4) x64 Intel(R) Core(TM) i5-5287U CPU @ 2.90GHz
Binaries:
Node: 12.18.2 - C:\Program Files\nodejs\node.EXE
npm: 6.14.5 - C:\Program Files\nodejs\npm.CMD
Languages:
Python: 3.8.4
Browsers:
Edge: 44.18362.449.0
npmPackages:
gatsby: ^2.21.40 => 2.21.40
gatsby-background-image: ^1.1.1 => 1.1.1
gatsby-image: ^2.2.37 => 2.4.6
gatsby-plugin-google-tagmanager: ^2.3.4 => 2.3.4
gatsby-plugin-instagram-embed: ^2.0.1 => 2.0.1
gatsby-plugin-manifest: ^2.2.34 => 2.4.10
gatsby-plugin-offline: ^3.0.30 => 3.2.8
gatsby-plugin-preact: ^3.3.2 => 3.3.2
gatsby-plugin-react-helmet: ^3.1.18 => 3.3.3
gatsby-plugin-remove-console: 0.0.2 => 0.0.2
gatsby-plugin-sharp: ^2.3.10 => 2.6.10
gatsby-plugin-twitter: ^2.3.3 => 2.3.3
gatsby-plugin-web-font-loader: ^1.0.4 => 1.0.4
gatsby-source-filesystem: ^2.1.43 => 2.3.10
gatsby-transformer-sharp: ^2.3.9 => 2.5.4
npmGlobalPackages:
gatsby-cli: 2.12.51

[sidharthachatterjee]

@tim-line Can you try running gatsby develop with NODE_OPTIONS=--max-old-space-size=8192

I suspect we aren't passing through node options to the child process (develop has a parent and child process now where the child is the main gatsby process which is hitting the allocation error)

[tim-line]

@sidharthachatterjee I've tried running:

node --max-old-space-size=8192 develop

And get the following error:

internal/modules/cjs/loader.js:969
  throw err;
  ^

Error: Cannot find module 'C:\projects\clientname\projectname\develop'
�[90m    at Function.Module._resolveFilename (internal/modules/cjs/loader.js:966:15)�[39m
�[90m    at Function.Module._load (internal/modules/cjs/loader.js:842:27)�[39m
�[90m    at Function.executeUserEntryPoint [as runMain] (internal/modules/run_main.js:71:12)�[39m
�[90m    at internal/main/run_main_module.js:17:47�[39m {
  code: �[32m'MODULE_NOT_FOUND'�[39m,
  requireStack: []
}

Am I missing something or running this incorrectly?

[sidharthachatterjee]

That's cause node --max-old-space-size=8192 develop is trying to call a file called develop that doesn't exist.

Try running NODE_OPTIONS=--max-old-space-size=8192 gatsby develop

[tim-line]

@sidharthachatterjee Thanks - appreciate this may not strictly be a gatsby issue. I've tried adding the following but still get the same error:

"scripts": {
    "develop": "NODE_OPTIONS=--max-old-space-size=8192 gatsby develop",
   ...
}

Have also tried running this direct from cmd prompt. Any further help much appreciated!

[tim-line]

@gatsbybot @sidharthachatterjee Any chance of reopening this as the solution isn't working for me. I'd also like to understand why this works fine before upgrading from 2.21.40. Thanks.

[vladar]

@tim-line Just to confirm - the build works on 2.21.40 and fails on 2.22.0 (the next version after 2.21.40)? If so, then the only related PR is #22759

Also is it possible to get access to your project somehow to reproduce? Or maybe you can prepare a minimum reproduction?

[MikeCastillo1]

Hello gatsby team, I have the same issue when upgrading from "^2.24.50", to "^2.24.53".

This is the stack trace at development mode

[============================]   10.299 s 6/6 100% Generating image thumbnails
[=========                   ]   9.095 s 7/21 33% run page queries

<--- Last few GCs --->

[39282:0x1041cb000]   218447 ms: Mark-sweep (reduce) 4091.4 (4100.9) -> 4090.4 (4100.9) MB, 4008.2 / 0.2 ms  (+ 1.0 ms in 1243 steps since start of marking, biggest step 0.0 ms, walltime since start of marking 5034 ms) (average mu = 0.290, current mu = 0.[39282:0x1041cb000]   223323 ms: Mark-sweep (reduce) 4091.4 (4100.9) -> 4090.4 (4101.2) MB, 4872.8 / 0.2 ms  (average mu = 0.155, current mu = 0.001) allocation failure scavenge might not succeed


<--- JS stacktrace --->

FATAL ERROR: Ineffective mark-compacts near heap limit Allocation failed - JavaScript heap out of memory
 1: 0x100bfea43 node::Abort() (.cold.1) [/usr/local/Cellar/node/14.5.0/bin/node]
 2: 0x10008609d node::FatalError(char const*, char const*) [/usr/local/Cellar/node/14.5.0/bin/node]
 3: 0x100086206 node::OnFatalError(char const*, char const*) [/usr/local/Cellar/node/14.5.0/bin/node]
 4: 0x10018cc95 v8::Utils::ReportOOMFailure(v8::internal::Isolate*, char const*, bool) [/usr/local/Cellar/node/14.5.0/bin/node]
 5: 0x10018cc3f v8::internal::V8::FatalProcessOutOfMemory(v8::internal::Isolate*, char const*, bool) [/usr/local/Cellar/node/14.5.0/bin/node]
 6: 0x1002b2ff1 v8::internal::Heap::FatalProcessOutOfMemory(char const*) [/usr/local/Cellar/node/14.5.0/bin/node]
 7: 0x1002b4354 v8::internal::Heap::MarkCompactPrologue() [/usr/local/Cellar/node/14.5.0/bin/node]
 8: 0x1002b1d3b v8::internal::Heap::PerformGarbageCollection(v8::internal::GarbageCollector, v8::GCCallbackFlags) [/usr/local/Cellar/node/14.5.0/bin/node]
 9: 0x1002b04e4 v8::internal::Heap::CollectGarbage(v8::internal::AllocationSpace, v8::internal::GarbageCollectionReason, v8::GCCallbackFlags) [/usr/local/Cellar/node/14.5.0/bin/node]
10: 0x1002b8654 v8::internal::Heap::AllocateRawWithLightRetrySlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/Cellar/node/14.5.0/bin/node]
11: 0x1002b86aa v8::internal::Heap::AllocateRawWithRetryOrFailSlowPath(int, v8::internal::AllocationType, v8::internal::AllocationOrigin, v8::internal::AllocationAlignment) [/usr/local/Cellar/node/14.5.0/bin/node]
12: 0x1002985a3 v8::internal::Factory::NewFillerObject(int, bool, v8::internal::AllocationType, v8::internal::AllocationOrigin) [/usr/local/Cellar/node/14.5.0/bin/node]
13: 0x10050a25f v8::internal::Runtime_AllocateInYoungGeneration(int, unsigned long*, v8::internal::Isolate*) [/usr/local/Cellar/node/14.5.0/bin/node]
14: 0x10077d659 Builtins_CEntry_Return1_DontSaveFPRegs_ArgvOnStack_NoBuiltinExit [/usr/local/Cellar/node/14.5.0/bin/node]
15: 0x3115704495de 
16: 0x3115704490f4 

and this on build is not progressing

[============================]   12.827 s 6/6 100% Generating image thumbnails
[=========                   ]   11.763 s 7/20 35% run page queries

[github-actions]

Hiya!

This issue has gone quiet. Spooky quiet. 👻

We get a lot of issues, so we currently close issues after 60 days of inactivity. It’s been at least 20 days since the last update here.
If we missed this issue or if you want to keep it open, please reply here.
As a friendly reminder: the best way to see this issue, or any other, fixed is to open a Pull Request. Check out gatsby.dev/contribute for more information about opening PRs, triaging issues, and contributing!

Thanks for being a part of the Gatsby community! 💪💜

[pvdz]

I'm going to close this issue as the topic is very generic and is bound to attract unrelated issues.

If you've come here to find a solution for an OOM ("Out Of Memory") issue, please try to bump the memory first (either NODE_OPTIONS=--max-old-space-size=8192 gatsby develop or node --max-old-space-size=8192 node_modules/.bin/gatsby develop and see if that works.

If you think you have a relatively small site / few images and getting an OOM, please open a new issue. Make sure to post a reproduction too and as much info as possible. Feel free to tag me into it.

Going to close this and lock it. Thank you 💜