Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Hermes Engine takes 2.5 to 19 times longer to process promises than Chakra #92

Open
AlexLablaiksSAP opened this issue Apr 6, 2022 · 7 comments · Fixed by #104
Open
Assignees
Labels

Comments

@AlexLablaiksSAP
Copy link

Problem Description

We have recently switched to Hermes to leverage Direct Debugging via VSCode. Unfortunately, we have been noticing that large datasets have been taking around 20 times longer in Hermes to process promises verses Chakra. Specifically, datasets that take 1.2 seconds with Chakra verses 25 seconds with Hermes.

I have been able to reproduce this with a simple React Native app available at hermes-promise-test, which uses version 0.68.0. Our metadata application often processes fields based on various rule sets. I have replicated this in the aforementioned repository by formatting the data set field values to uppercase or lowercase. The results are as follows:

  1. Flat (one promise per field) of 1,000 objects: 73.29 ms vs 4.71 ms or 15.5 times longer for Hermes.
  2. Flat of 10,000 objects: 748 ms vs 23 ms or 32 times longer for Hermes.
  3. Nested (each object has a Promise.all() for every field (6)) of 1,000 objects: 379 ms vs 9 ms or 41 times longer for Hermes.
  4. Nested of 10,000 objects: 23626 ms vs 145 ms or 163 times longer for Hermes.

Steps To Reproduce

  1. Checkout hermes-promise-test
  2. Observe Hermes and Chakra differences by toggling between the two engines by pressing Ctrl + Shift + D and selecting "<Enable/Disable> Remote JS Debugging".
  3. Choose Mock Data with Flat Promises or Mock Data with Nested Promises
  4. Click either Get 1k Object Cells or Get 10k Object Cells to populate data.
  5. Click the Format button to process the data set. Note that times will appear in the console log.

Expected Results

Hermes to perform similar or better than Chakra.

CLI version

7.0.3

Environment

System:
    OS: Windows 10 10.0.19042
    CPU: (16) x64 Intel(R) Core(TM) i9-10885H CPU @ 2.40GHz
    Memory: 16.96 GB / 31.75 GB
  Binaries:
    Node: 16.14.0 - C:\Program Files\nodejs\node.EXE
    Yarn: 1.22.15 - C:\Program Files (x86)\Yarn\bin\yarn.CMD
    npm: 8.3.1 - C:\Program Files\nodejs\npm.CMD
    Watchman: Not Found
  SDKs:
    Android SDK:
      API Levels: 28, 29, 30
      Build Tools: 28.0.3, 29.0.2, 30.0.3, 31.0.0
      System Images: android-30 | Google APIs Intel x86 Atom
      Android NDK: Not Found
    Windows SDK:
      AllowDevelopmentWithoutDevLicense: Enabled
      AllowAllTrustedApps: Enabled
      Versions: 10.0.16299.0, 10.0.18362.0, 10.0.19041.0
  IDEs:
    Android Studio: Version  4.2.0.0 AI-202.7660.26.42.7351085
    Visual Studio: 16.11.32228.343 (Visual Studio Enterprise 2019)
  Languages:
    Java: 11.0.10 - C:\Program Files\Microsoft\jdk-11.0.10.9-hotspot\bin\javac.EXE
  npmPackages:
    @react-native-community/cli: Not Found
    react: 17.0.2 => 17.0.2
    react-native: 0.68.0 => 0.68.0
    react-native-windows: 0.68.0 => 0.68.0
  npmGlobalPackages:
    *react-native*: Not Found

Target Platform Version

10.0.19041

Target Device(s)

Desktop

Visual Studio Version

Visual Studio 2019

Build Configuration

Debug

Snack, code example, screenshot, or link to a repository

https://github.com/AlexLablaiksSAP/hermes-promise-test

@AlexLablaiksSAP AlexLablaiksSAP added the bug Something isn't working label Apr 6, 2022
@ghost ghost added the Needs: triage 🔍 label Apr 6, 2022
@AlexLablaiksSAP
Copy link
Author

hermes-promise-test.xlsx

@NickGerleman
Copy link

NickGerleman commented Apr 6, 2022

From your steps, you mentioned opening the RNW developer menu. Are you running a debug build of your application?

Chakra exposes it's APIs to react-native-windows in the form of ABI-safe C functions. It lets RNW debug builds use a release version of Chakra. The Hermes boundary is not yet ABI safe, and MSVC has different debug snd release ABIs. This currently forces RNW to use an "optimized debug build" of Hermes, which performs worse than its release optimized build.

This constraint go away I think, with work to move JS engine interface to be ABI safe. But right now Hermes direct debugging, like web debugging, is not representative of release build performance. Debug-time developer experience is still super important though. Did switching to Hermes direct debugging degrade that for your app?

@acoates-ms
Copy link
Collaborator

Also from your steps you are not comparing Chakra vs Hermes at all. When you enable remote debugging, you are running the JS inside a webbrowser (either Edge or Chrome probably) - both of which use V8. When running the JS in a browser you are running inside a very different JS environment including a completely different promise implementation. There are so many differences between how things run in "Remote Debugging" env, that it should not be used for any kind of performance testing.

@AlexLablaiksSAP
Copy link
Author

AlexLablaiksSAP commented Apr 6, 2022

From your steps, you mentioned opening the RNW developer menu. Are you running a debug build of your application?

Chakra exposes it's APIs to react-native-windows in the form of ABI-safe C functions. It lets RNW debug builds use a release version of Chakra. The Hermes boundary is not yet ABI safe, and MSVC has different debug snd release ABIs. This currently forces RNW to use an "optimized debug build" of Hermes, which performs worse than its release optimized build.

This constraint go away I think, with work to move JS engine interface to be ABI safe. But right now Hermes direct debugging, like web debugging, is not representative of release build performance. Debug-time developer experience is still super important though. Did switching to Hermes direct debugging degrade that for your app?

Yes, all tests were performed with a Debug build initially and were essentially live switched using the developer menu.

For our actual application, which my test repository is imitating, we do indeed have about a 20 second load screen time which is similar to the 10K Nested Promise test in the referenced repository.

I have gone back and rebuilt and tested in a Release build. The performance of Hermes in Release was still worse than webdebugging. It's still about 8 seconds for a nested 10K promise test. The others are under 1 second but still have the worst performance. I have updated the application to alert with the measure and have updated the spreadsheet with the data.
hermes-promise-test.xlsx

@AlexLablaiksSAP
Copy link
Author

Also from your steps you are not comparing Chakra vs Hermes at all. When you enable remote debugging, you are running the JS inside a webbrowser (either Edge or Chrome probably) - both of which use V8. When running the JS in a browser you are running inside a very different JS environment including a completely different promise implementation. There are so many differences between how things run in "Remote Debugging" env, that it should not be used for any kind of performance testing.

I can go ahead and provide you with updated results from Chakra in release for the repository in question.

@AlexLablaiksSAP
Copy link
Author

I have tested Chakra with both Debug and Release. Results are as follows:
For Debug, Hermes takes 3.5 to 19 times longer than Chakra.
For Release, Hermes takes 2.5 to 7.7 times longer than Chakra.

Important to note; for the larger Nested Promise tests, Chakra Debug outperforms Hermes in Release.
hermes-promise-test.xlsx

@AlexLablaiksSAP AlexLablaiksSAP changed the title Hermes Engine takes 15 to 160 times longer to process promises than Chakra Hermes Engine takes 2.5 to 19 times longer to process promises than Chakra Apr 6, 2022
@chrisglein chrisglein transferred this issue from microsoft/react-native-windows Apr 7, 2022
tudorms added a commit to tudorms/hermes-windows that referenced this issue Jun 2, 2022
@tudorms tudorms mentioned this issue Jun 2, 2022
tudorms added a commit that referenced this issue Jun 3, 2022
Disable Stats Timer (addresses #92)
Include revision as part of product version in addition to suffix (addresses #91)
Include license file in NuGet (addresses #98)
Add Source Link to linker (addresses #102)
@tudorms tudorms mentioned this issue Jun 4, 2022
@ghost ghost added the Status: In PR label Jun 4, 2022
@ghost ghost closed this as completed in #104 Jun 4, 2022
ghost pushed a commit that referenced this issue Jun 4, 2022
- Disable Stats Timer (fixes #92)
- Include revision as part of product version in addition to suffix (fixes #91)
- Include license file in NuGet (fixes #98)
- Add Source Link to linker (fixes #102)

This is ported from the v0.11 branch. We bump the version up to 0.12 (even though no such version exists in upstream) to signal we're no longer compatible with 0.11 (and RN 0.68).
@ghost ghost added Status: Fixed and removed Status: In PR labels Jun 4, 2022
@ghost ghost locked as resolved and limited conversation to collaborators Jul 4, 2022
@vmoroz vmoroz reopened this Jun 5, 2023
@vmoroz
Copy link
Member

vmoroz commented Jun 5, 2023

@AlexLablaiksSAP I am reopening the issue.
As we discussed with your team today, I clearly see the same issue playing with your repro.
From what I see in the WPA (Windows Perf Analysis tooling), where are no significant delays in the JS thread. The most of time is spent inside of the Hermes interpreter. The function that seems to be the most expensive is the Array.indexOf().
image

The next steps:

  • Discuss this issue with Meta's Hermes team
  • I would like to see the Hermes CPU sampling profiler results. I need to use the latest Hermes bit for it.

@vmoroz vmoroz self-assigned this Jun 5, 2023
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
Projects
None yet
Development

Successfully merging a pull request may close this issue.

4 participants