RangeError: Maximum call stack size exceeded when require/load very big json file

[dongyuwei]

https://github.com/dongyuwei/tiny_english_dictionary/blob/hermes/trie-service.js#L2
const words = require('./words_with_frequency_and_translation_and_ipa.json'); will throw error:

RangeError: Maximum call stack size exceeded, js engine:hermes

The json file size is 21M, while its data structure is very simple.

Steps to reproduce the bug:

1. yarn start
2. start an android emulator
3. yarn android

Tested with yarn 1.17.3 and nodejs 12.12.0 on Mac 10.14.5.

[dongyuwei]

Exception call stack:

[tmikov]

Thanks for reporting this. It seems like we have a problem with compiling large JSON files. We are looking into it and will post an update here.

Meanwhile, as a workaround can you parse as a string with JSON.parse()?

[dongyuwei]

Thanks for your reply, I'll try and let you know today. On road now.

[dongyuwei]

I just verified JSON.parse works. See https://github.com/dongyuwei/tiny_english_dictionary/blob/hermes/trie-service.js#L18, I use react-native-local-resource to load the stringified json txt, then use JSON.parse to decode it.

[sofiageo]

In my case I have a 14MB file which builds fine in debug mode but Hermes hangs when trying to optimize. I had to turn off hermes optimizations for the build to complete. Let me know if I can help to somehow debug this for you.

[dongyuwei]

There is a unit test to parse json: /unittests/Parser/JSONParserTest.cpp, but I don't know how to run the test.

@tmikov can you modify and run the test with my big json file words_with_frequency_and_translation_and_ipa.json? From the call stack of the exception, can't see any direct relationship with hermes.

[tmikov]

We investigated this when it was reported and we believe we understand the bug. We haven't forgotten it, and we have been workin to address it, but progress has been slow because the fix is non-trivial. There are two separate problems:

• Without optimizations, the generated code for large nested object literals consume inordinate amount of register stack, causing the stack overflow exception.
• With optimizations, the stack usage is much better, but one particular optimization pass exhibits degenerate behavior on the same kind of object literals.

The workaround is to use JSON.parse() with a string or a resource.

I will look into escalating this, so the fix can land sooner.

[aprilmintacpineda]

This happened to me when I upgraded my react-native https://react-native-community.github.io/upgrade-helper/?from=0.63.4&to=0.64.0

 WARN  Require cycle: node_modules/core-js/internals/microtask.js -> node_modules/core-js/internals/microtask.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Blob.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/XMLHttpRequest.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Fetch.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 ERROR  RangeError: Maximum call stack size exceeded, js engine: hermes
 LOG  Running "ny_app" with {"rootTag":1}
 ERROR  Invariant Violation: "ny_app" has not been registered. This can happen if:
* Metro (the local dev server) is run from the wrong folder. Check if Metro is running, stop it and restart it in the current project.
* A module failed to load due to an error and `AppRegistry.registerComponent` wasn't called., js engine: hermes

Happens both on android and ios.

[melroyd16]

This happened to me when I upgraded my react-native https://react-native-community.github.io/upgrade-helper/?from=0.63.4&to=0.64.0

 WARN  Require cycle: node_modules/core-js/internals/microtask.js -> node_modules/core-js/internals/microtask.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Blob.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/XMLHttpRequest.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Fetch.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 ERROR  RangeError: Maximum call stack size exceeded, js engine: hermes
 LOG  Running "ny_app" with {"rootTag":1}
 ERROR  Invariant Violation: "ny_app" has not been registered. This can happen if:
* Metro (the local dev server) is run from the wrong folder. Check if Metro is running, stop it and restart it in the current project.
* A module failed to load due to an error and `AppRegistry.registerComponent` wasn't called., js engine: hermes

Happens both on android and ios.

Setting inlineRequires: false in metro.config.js fixed it for me

[chensu-chime]

We investigated this when it was reported and we believe we understand the bug. We haven't forgotten it, and we have been workin to address it, but progress has been slow because the fix is non-trivial. There are two separate problems:

• Without optimizations, the generated code for large nested object literals consume inordinate amount of register stack, causing the stack overflow exception.
• With optimizations, the stack usage is much better, but one particular optimization pass exhibits degenerate behavior on the same kind of object literals.

The workaround is to use JSON.parse() with a string or a resource.

I will look into escalating this, so the fix can land sooner.

Is there any update on the fix?

[marcibk]

I have a 13mb JSON file which should be loaded locally. I tried basically everything, but it always gives me this error... Can someone help me? Any working work-around for this? Or a fix somehow? Thank you... @tmikov

[tmikov]

@traingethermarc the workaround is to use JSON.parse().

[aprilmintacpineda]

This happened to me when I upgraded my react-native https://react-native-community.github.io/upgrade-helper/?from=0.63.4&to=0.64.0

 WARN  Require cycle: node_modules/core-js/internals/microtask.js -> node_modules/core-js/internals/microtask.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Blob.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/XMLHttpRequest.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Fetch.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 ERROR  RangeError: Maximum call stack size exceeded, js engine: hermes
 LOG  Running "ny_app" with {"rootTag":1}
 ERROR  Invariant Violation: "ny_app" has not been registered. This can happen if:
* Metro (the local dev server) is run from the wrong folder. Check if Metro is running, stop it and restart it in the current project.
* A module failed to load due to an error and `AppRegistry.registerComponent` wasn't called., js engine: hermes

Happens both on android and ios.

This is happening to one of the apps I'm working on (1) BUT it does not import any big JSON files. On the other hand, the other app I'm working on (2) doesn't experience this, could it be related to auth0? because my former uses auth0 but the latter does not.

[dongdyang]

it has been two years. And this issue still occurs for the latest 0.9 version hermes.

[cristianoccazinsp]

What's hermes' max call stack size? I'm getting max call stack size errors for recursive functions as deep as just 60 calls.

[tmikov]

@cristianoccazinsp For JS to JS recursion, the maximum depth is dynamic and depends on the size of every frame and the size of the register stack, which is configurable. In the default configuration, the following code performs 55,000 recursive invocations without optimization, and 87,000 with optimization:

let level = 0;
function foo() {
    ++level;
    foo();
}

try {
    foo();
} catch (e) {
    print("Level", level);
    print(e);
}

Hermes has a separate fixed stack limit for native calls, which is set to 128. So, if you change the recursive invocation from foo() to foo.call(), it throws after 128 recursive calls.

[cristianoccazinsp]

Native stack errors is just the crash I'm facing right now. Sigh, 128 is so small! This used to work fine with JSC and now blows up with Hermes. Sadly, the code is for an HTML parsing library that relies on recursion so 128 will definitely cause issues as HTMLs could be huge.

[tmikov]

Keep in mind that the low number is only for cases involving recursion through a native function. The number is deliberately very conservative to guarantee that a native stack overflow will never occur. It is even lower in some build modes, because the native stack (which is not controlled by Hermes) can and does overflow quite easily.

The limit can be increased by changing this line:

hermes/include/hermes/VM/Runtime.h

Line 905 in 2c66a23

384

There is some work in progress for obtaining and checking the actual stack size, which on most cases will result in significantly increased depth, but no promises when it will land.

[cristianoccazinsp]

I'm not entirely sure why I get native stack depth errors even though the recursion happens entirely in JS. Does calls to .map and "native" methods count towards the native stack?

[tmikov]

Yes. Most "standard library" functions are implemented in C++ and unfortunately count as native.

The best way to address this is to check whether the native stack is actually overflowing, instead of setting artificial hard limits. I will see if I can finish that work.

[cristianoccazinsp]

@tmikov I replaced all .map calls with simple for a in b loops and the recursion errors went away (even though I'm still using recursive calls). Funny thing is, the moment the screen gets unmounted, another native stack depth error occurs from a totally different place (react-navigation stack and animations).

[aikewoody]

With this comment I want to raise awareness of this issue and show that there is a high demand for it to be fixed. (Even though there is a workaround to import as string and then use JSON.parse)

[Waltari10]

Hi, I'm trying to upgrade to Hermes in our RN app and I'm getting the ERROR RangeError: Maximum call stack size exceeded (native stack depth), js engine: hermes. No big JSON files. I guess this is just Javascript code accessing the bridge too often?

Is the default max stack size going to be increased in RN 0.72?

[mikeduminy]

@Waltari10 the fix has landed in RN 0.71.4 and will likely be cherry picked into RN 0.70.9. It will also be in RN 0.72.

[Waltari10]

@Waltari10 the fix has landed in RN 0.71.4 and will likely be cherry picked into RN 0.70.9. It will also be in RN 0.72.

Thanks for the info! Curiously we are already using 0.71.4 in our app though and still getting the error 🤔

[mikeduminy]

Disclaimer: I'm not an expert.

Oh I just noticed that your error message includes (native stack depth) which is a different error. The one that was fixed in RN 0.71.4 was a JS Register max stack size error, but yours is about the native stack size (which is much lower). You can find more information in this comment.

No big JSON files. I guess this is just Javascript code accessing the bridge too often?

Since this is a native stack issue I guess that neither JSON nor frequent bridge access is the problem. What does your stack trace look like?

Have a look at the stack trace in this comment. Notice the multiple lines that say: at apply (native). This indicates that a native call is being made. Note: this is not a react native call, but rather what the runtime considers a native call, which would be things like Function.prototype.call, Function.prototype.apply, or (maybe) Array.prototype.map.

Maybe the transpilation step of your builds is adding too many of these native calls to functions that are called in a deeply-nested way. You could see if any changes to your babel settings could fix it. Or perhaps you need to hunt down what changed in your codebase from when it was working.

[aurangs7]

This happened to me when I upgraded my react-native https://react-native-community.github.io/upgrade-helper/?from=0.63.4&to=0.64.0

 WARN  Require cycle: node_modules/core-js/internals/microtask.js -> node_modules/core-js/internals/microtask.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Blob.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/XMLHttpRequest.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 WARN  Require cycle: node_modules/rn-fetch-blob/index.js -> node_modules/rn-fetch-blob/polyfill/index.js -> node_modules/rn-fetch-blob/polyfill/Fetch.js -> node_modules/rn-fetch-blob/index.js

Require cycles are allowed, but can result in uninitialized values. Consider refactoring to remove the need for a cycle.
 ERROR  RangeError: Maximum call stack size exceeded, js engine: hermes
 LOG  Running "ny_app" with {"rootTag":1}
 ERROR  Invariant Violation: "ny_app" has not been registered. This can happen if:
* Metro (the local dev server) is run from the wrong folder. Check if Metro is running, stop it and restart it in the current project.
* A module failed to load due to an error and `AppRegistry.registerComponent` wasn't called., js engine: hermes

Happens both on android and ios.

Setting inlineRequires: false in metro.config.js fixed it for me

I am also getting this error in Android (iOS working fine), this happened suddenly.
It was working fine till yesterday.

inlineRequires: false => is already added in file

Any solution?

[antondalgren]

Hi,

Been struck by this issue too when migrating from RN 0.69 to 0.70.9 and enabling hermes in the process.

Would this limitation of the call stack on Function.call() also be applicable to the usage of Array.prototype.reduce? As the implementation of the reduce function is calling the Callable::executeCall4 which in turn uses ScopedNativeCallFrame.

hermes/lib/VM/JSLib/Array.cpp

Line 3276 in d1be3ff

auto callRes = Callable::executeCall4(

hermes/lib/VM/Callable.cpp

Line 296 in 081b8ac

CallResult<PseudoHandle<>> Callable::executeCall4(

We are essentially using AJV for JSON validation, and that library is using recursion within the call and reduce method.

Also noted that Function.apply() is also limited by this call stack.

hermes/lib/VM/JSLib/Function.cpp

Line 211 in d1be3ff

ScopedNativeCallFrame newFrame{runtime, 0, *func, false, args.getArg(0)};

[tmikov]

FWIW, this has been addressed in the next version of Hermes, but unfortunately we do not yet have a timeline for release or for backporting.

[alvarovillafane]

Hi @tmikov,

Would you mind helping me to clarify some things I'm a bit confused about?

I just got hit by the RangeError: Maximum call stack size exceeded (native stack depth) issue while working on my app.

This seems to be different from the original issue of this thread that was fixed in #135 (comment)
I'm correct? fixing this issue will cover both cases or we should have another issue to track the "native stack depth" one?

In any case, my issue is that I'm using a library that needs to do recursive work where it uses apply/call functions multiple times.
I understand that the functions are converted to native calls and there is stack limit of 128. On certain situations I hit the limit and got the error.

It works fine if I use JavaScriptCore as the engine, I did a rough test in the stack limit seems to be around ~2738 when using JSC.

Is there a way to increase the limit when using Hermes or any other workaround?
Unfortunately, I can't change a 3rd party library that my app depends on.

Here is a repo with issue: https://github.com/vmalvaro/hermes-range-error

Thanks!

[tmikov]

@vmalvaro the limit can be increased by changing Hermes as described in this comment.

[alvarovillafane]

@tmikov thanks for the quick response, I'm a bit of a newbie with Hermes so I'd appreciate your guidance.

This #135 means that I have to do that change in hermes/include/hermes/VM/Runtime.h, then a custom build of Hermes + react native from source as described here https://hermesengine.dev/docs/react-native-integration?

[neildhar]

@vmalvaro

custom build of Hermes + react native from source as described here

Those instructions are unfortunately no longer accurate for newer versions of RN, which have changed how Hermes is consumed. If you're on an RN version that is 0.69+, you can set an environment variable to point RN to the source directory where you want it to get Hermes from. See https://github.com/facebook/react-native/blob/fd9e295befcd8781190ec26a6a2fc4ef39fb1c15/packages/react-native/ReactAndroid/hermes-engine/build.gradle#L42.

[Villar74]

Get this issue when disabling js DEV mode

 System:
    OS: macOS 13.4
    CPU: (12) arm64 Apple M2 Pro
  Binaries:
    Node: 20.0.0 - /opt/homebrew/bin/node
    Yarn: 1.22.19 - /opt/homebrew/bin/yarn
    npm: 9.6.4 - /opt/homebrew/bin/npm
    Watchman: 2023.05.01.00 - /opt/homebrew/bin/watchman
  SDKs:
    iOS SDK:
      Platforms: DriverKit 22.4, iOS 16.4, macOS 13.3, tvOS 16.4, watchOS 9.4
    Android SDK:
      API Levels: 31, 33
      Build Tools: 30.0.2, 30.0.3, 31.0.0, 33.0.0, 33.0.2
      System Images: android-33 | Google APIs ARM 64 v8a
  IDEs:
    Android Studio: Flamingo 2022.2.1 Patch 1 Flamingo 2022.2.1 Patch 1
    Xcode: 14.3/14E222b - /usr/bin/xcodebuild
  npmPackages:
    react: 18.2.0 => 18.2.0 
    react-native: 0.71.8 => 0.71.8 

[tmikov]

@Villar74 is your problem related to loading JSON, which is the topic of this specific issue?

If not, and you believe there is a bug in Hermes, please create a new issue, preferably with more details, including source reproducing the problem. It is almost impossible to diagnose a problem from a screenshot of a stack trace of a compiled bundle.

[tmikov]

Finally, this long standing problem has been fixed in 3213794 and should be available in the next release of RN.

[Yashtoddle]

Hey tmikov could you please send this update in the coming patch as I needed this issue to be resolved in our codebase

[tmikov]

@Yashtoddle I don't understand what you mean. Send what where?

[Yashtoddle]

Sorry for not explaining the stuff clearly @tmikov actually we are using Hermes in our project as the engine and now we are facing the maximum call stack size exceded error in the android app and this error is now being resolved it would be great if you send 3213794
in the coming patch of the react native as we need this urgently in our project

[tmikov]

@Yashtoddle we don't really control which changes are picked into each patch releases. You also need to be more specific about the version where you want this.

In any case I think you can request a pick here: https://github.com/reactwg/react-native-releases/discussions .

This is not a trivial bugfix, it is several commits including 18aa617 .

@neildhar @cortinico what is the feasibility of picking this?

[neildhar]

@Yashtoddle this will be included in the upcoming RN 0.73. The feasibility of picking this fix into an earlier release will likely depend heavily on which RN version you are currently on.