tools: port js2c.py to C++ #46997
Closed
tools: port js2c.py to C++ #46997
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
|
Review requested:
|
nodejs-github-bot
added
build
|
cc @nodejs/build @nodejs/build-files |
|
@joyeecheung do you mind if I take over this PR and rebase it? I have some ideas I would like to build on top of this :) |
This makes it easier to use third-party dependencies in this tool (e.g. adding compression using algorithms not available in Python). It is also much faster - locally js2c.py takes ~1.5s to generate the output whereas this version takes ~0.1s - and consumes less memory (~110MB v.s. 66MB). This also modifies the js2c.py a bit to simplify the output, making it easier to compare with one generated by the C++ version. Locally the output from the two are identical. We'll remove js2c.py in a subsequent commit when the C++ version is used by default.
|
I actually had the rebase locally, and was just waiting for the two other js2c changes to land before I could push it (and also, waiting for reviews, which this still doesn't have...so ping @nodejs/build @nodejs/build-files again..) |
anonrig
approved these changes
May 21, 2023
github-actions
bot
removed
the
request-ci
|
cc @nodejs/cpp-reviewers |
|
@joyeecheung can we merge this PR? |
|
Landed in ceb2a21...4da7bc9 |
joyeecheung
added a commit
that referenced
this pull request
May 24, 2023
This makes it easier to use third-party dependencies in this tool (e.g. adding compression using algorithms not available in Python). It is also much faster - locally js2c.py takes ~1.5s to generate the output whereas this version takes ~0.1s - and consumes less memory (~110MB v.s. 66MB). This also modifies the js2c.py a bit to simplify the output, making it easier to compare with one generated by the C++ version. Locally the output from the two are identical. We'll remove js2c.py in a subsequent commit when the C++ version is used by default. PR-URL: #46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
joyeecheung
added a commit
that referenced
this pull request
May 24, 2023
PR-URL: #46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
targos
pushed a commit
that referenced
this pull request
May 30, 2023
This makes it easier to use third-party dependencies in this tool (e.g. adding compression using algorithms not available in Python). It is also much faster - locally js2c.py takes ~1.5s to generate the output whereas this version takes ~0.1s - and consumes less memory (~110MB v.s. 66MB). This also modifies the js2c.py a bit to simplify the output, making it easier to compare with one generated by the C++ version. Locally the output from the two are identical. We'll remove js2c.py in a subsequent commit when the C++ version is used by default. PR-URL: #46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
targos
pushed a commit
that referenced
this pull request
May 30, 2023
PR-URL: #46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
danielleadams
added
the
backport-requested-v18.x
Ceres6
pushed a commit
to Ceres6/node
that referenced
this pull request
Aug 14, 2023
This makes it easier to use third-party dependencies in this tool (e.g. adding compression using algorithms not available in Python). It is also much faster - locally js2c.py takes ~1.5s to generate the output whereas this version takes ~0.1s - and consumes less memory (~110MB v.s. 66MB). This also modifies the js2c.py a bit to simplify the output, making it easier to compare with one generated by the C++ version. Locally the output from the two are identical. We'll remove js2c.py in a subsequent commit when the C++ version is used by default. PR-URL: nodejs#46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Ceres6
pushed a commit
to Ceres6/node
that referenced
this pull request
Aug 14, 2023
PR-URL: nodejs#46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Ceres6
pushed a commit
to Ceres6/node
that referenced
this pull request
Aug 14, 2023
This makes it easier to use third-party dependencies in this tool (e.g. adding compression using algorithms not available in Python). It is also much faster - locally js2c.py takes ~1.5s to generate the output whereas this version takes ~0.1s - and consumes less memory (~110MB v.s. 66MB). This also modifies the js2c.py a bit to simplify the output, making it easier to compare with one generated by the C++ version. Locally the output from the two are identical. We'll remove js2c.py in a subsequent commit when the C++ version is used by default. PR-URL: nodejs#46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Ceres6
pushed a commit
to Ceres6/node
that referenced
this pull request
Aug 14, 2023
PR-URL: nodejs#46997 Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
3 tasks
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 2, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 6, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 14, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 15, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 16, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 21, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 22, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 28, 2023
codebytere
added a commit
to electron/electron
that referenced
this pull request
Nov 29, 2023
jkleinsc
pushed a commit
to electron/electron
that referenced
this pull request
Nov 30, 2023
* chore: upgrade to Node.js v20 * src: allow embedders to override NODE_MODULE_VERSION nodejs/node#49279 * src: fix missing trailing , nodejs/node#46909 * src,tools: initialize cppgc nodejs/node#45704 * tools: allow passing absolute path of config.gypi in js2c nodejs/node#49162 * tools: port js2c.py to C++ nodejs/node#46997 * doc,lib: disambiguate the old term, NativeModule nodejs/node#45673 * chore: fixup Node.js BSSL tests * nodejs/node#49492 * nodejs/node#44498 * deps: upgrade to libuv 1.45.0 nodejs/node#48078 * deps: update V8 to 10.7 nodejs/node#44741 * test: use gcUntil() in test-v8-serialize-leak nodejs/node#49168 * module: make CJS load from ESM loader nodejs/node#47999 * src: make BuiltinLoader threadsafe and non-global nodejs/node#45942 * chore: address changes to CJS/ESM loading * module: make CJS load from ESM loader (nodejs/node#47999) * lib: improve esm resolve performance (nodejs/node#46652) * bootstrap: optimize modules loaded in the built-in snapshot nodejs/node#45849 * test: mark test-runner-output as flaky nodejs/node#49854 * lib: lazy-load deps in modules/run_main.js nodejs/node#45849 * url: use private properties for brand check nodejs/node#46904 * test: refactor `test-node-output-errors` nodejs/node#48992 * assert: deprecate callTracker nodejs/node#47740 * src: cast v8::Object::GetInternalField() return value to v8::Value nodejs/node#48943 * test: adapt test-v8-stats for V8 update nodejs/node#45230 * tls: ensure TLS Sockets are closed if the underlying wrap closes nodejs/node#49327 * test: deflake test-tls-socket-close nodejs/node#49575 * net: fix crash due to simultaneous close/shutdown on JS Stream Sockets nodejs/node#49400 * net: use asserts in JS Socket Stream to catch races in future nodejs/node#49400 * lib: fix BroadcastChannel initialization location nodejs/node#46864 * src: create BaseObject with node::Realm nodejs/node#44348 * src: implement DataQueue and non-memory resident Blob nodejs/node#45258 * sea: add support for V8 bytecode-only caching nodejs/node#48191 * chore: fixup patch indices * gyp: put filenames in variables nodejs/node#46965 * build: modify js2c.py into GN executable * fix: (WIP) handle string replacement of fs -> original-fs * [v20.x] backport vm-related memory fixes nodejs/node#49874 * src: make BuiltinLoader threadsafe and non-global nodejs/node#45942 * src: avoid copying string in fs_permission nodejs/node#47746 * look upon my works ye mighty and dispair * chore: patch cleanup * [api] Remove AllCan Read/Write https://chromium-review.googlesource.com/c/v8/v8/+/5006387 * fix: missing include for NODE_EXTERN * chore: fixup patch indices * fix: fail properly when js2c fails in Node.js * build: fix js2c root_gen_dir * fix: lib/fs.js -> lib/original-fs.js * build: fix original-fs file xforms * fixup! module: make CJS load from ESM loader * build: get rid of CppHeap for now * build: add patch to prevent extra fs lookup on esm load * build: greatly simplify js2c modifications Moves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c * chore: update to handle moved internal/modules/helpers file * test: update @types/node test * feat: enable preventing cppgc heap creation * feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler * fix: no cppgc initialization in the renderer * gyp: put filenames in variables nodejs/node#46965 * test: disable single executable tests * fix: nan tests failing on node headers missing file * tls,http2: send fatal alert on ALPN mismatch nodejs/node#44031 * test: disable snapshot tests * nodejs/node#47887 * nodejs/node#49684 * nodejs/node#44193 * build: use deps/v8 for v8/tools Node.js hard depends on these in their builtins * test: fix edge snapshot stack traces nodejs/node#49659 * build: remove js2c //base dep * build: use electron_js2c_toolchain to build node_js2c * fix: don't create SafeSet outside packageResolve Fixes failure in parallel/test-require-delete-array-iterator: === release test-require-delete-array-iterator === Path: parallel/test-require-delete-array-iterator node:internal/per_context/primordials:426 constructor(i) { super(i); } // eslint-disable-line no-useless-constructor ^ TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator)) at new Set (<anonymous>) at new SafeSet (node:internal/per_context/primordials:426:22) * fix: failing crashReporter tests on Linux These were failing because our change from node::InitializeNodeWithArgs to node::InitializeOncePerProcess meant that we now inadvertently called PlatformInit, which reset signal handling. This meant that our intentional crash function ElectronBindings::Crash no longer worked and the renderer process no longer crashed when process.crash() was called. We don't want to use Node.js' default signal handling in the renderer process, so we disable it by passing kNoDefaultSignalHandling to node::InitializeOncePerProcess. * build: only create cppgc heap on non-32 bit platforms * chore: clean up util:CompileAndCall * src: fix compatility with upcoming V8 12.1 APIs nodejs/node#50709 * fix: use thread_local BuiltinLoader * chore: fixup v8 patch indices --------- Co-authored-by: Keeley Hammond <vertedinde@electronjs.org> Co-authored-by: Samuel Attard <marshallofsound@electronjs.org>
MrHuangJser
pushed a commit
to MrHuangJser/electron
that referenced
this pull request
Dec 11, 2023
* chore: upgrade to Node.js v20 * src: allow embedders to override NODE_MODULE_VERSION nodejs/node#49279 * src: fix missing trailing , nodejs/node#46909 * src,tools: initialize cppgc nodejs/node#45704 * tools: allow passing absolute path of config.gypi in js2c nodejs/node#49162 * tools: port js2c.py to C++ nodejs/node#46997 * doc,lib: disambiguate the old term, NativeModule nodejs/node#45673 * chore: fixup Node.js BSSL tests * nodejs/node#49492 * nodejs/node#44498 * deps: upgrade to libuv 1.45.0 nodejs/node#48078 * deps: update V8 to 10.7 nodejs/node#44741 * test: use gcUntil() in test-v8-serialize-leak nodejs/node#49168 * module: make CJS load from ESM loader nodejs/node#47999 * src: make BuiltinLoader threadsafe and non-global nodejs/node#45942 * chore: address changes to CJS/ESM loading * module: make CJS load from ESM loader (nodejs/node#47999) * lib: improve esm resolve performance (nodejs/node#46652) * bootstrap: optimize modules loaded in the built-in snapshot nodejs/node#45849 * test: mark test-runner-output as flaky nodejs/node#49854 * lib: lazy-load deps in modules/run_main.js nodejs/node#45849 * url: use private properties for brand check nodejs/node#46904 * test: refactor `test-node-output-errors` nodejs/node#48992 * assert: deprecate callTracker nodejs/node#47740 * src: cast v8::Object::GetInternalField() return value to v8::Value nodejs/node#48943 * test: adapt test-v8-stats for V8 update nodejs/node#45230 * tls: ensure TLS Sockets are closed if the underlying wrap closes nodejs/node#49327 * test: deflake test-tls-socket-close nodejs/node#49575 * net: fix crash due to simultaneous close/shutdown on JS Stream Sockets nodejs/node#49400 * net: use asserts in JS Socket Stream to catch races in future nodejs/node#49400 * lib: fix BroadcastChannel initialization location nodejs/node#46864 * src: create BaseObject with node::Realm nodejs/node#44348 * src: implement DataQueue and non-memory resident Blob nodejs/node#45258 * sea: add support for V8 bytecode-only caching nodejs/node#48191 * chore: fixup patch indices * gyp: put filenames in variables nodejs/node#46965 * build: modify js2c.py into GN executable * fix: (WIP) handle string replacement of fs -> original-fs * [v20.x] backport vm-related memory fixes nodejs/node#49874 * src: make BuiltinLoader threadsafe and non-global nodejs/node#45942 * src: avoid copying string in fs_permission nodejs/node#47746 * look upon my works ye mighty and dispair * chore: patch cleanup * [api] Remove AllCan Read/Write https://chromium-review.googlesource.com/c/v8/v8/+/5006387 * fix: missing include for NODE_EXTERN * chore: fixup patch indices * fix: fail properly when js2c fails in Node.js * build: fix js2c root_gen_dir * fix: lib/fs.js -> lib/original-fs.js * build: fix original-fs file xforms * fixup! module: make CJS load from ESM loader * build: get rid of CppHeap for now * build: add patch to prevent extra fs lookup on esm load * build: greatly simplify js2c modifications Moves our original-fs modifications back into a super simple python script action, wires up the output of that action into our call to js2c * chore: update to handle moved internal/modules/helpers file * test: update @types/node test * feat: enable preventing cppgc heap creation * feat: optionally prevent calling V8::EnableWebAssemblyTrapHandler * fix: no cppgc initialization in the renderer * gyp: put filenames in variables nodejs/node#46965 * test: disable single executable tests * fix: nan tests failing on node headers missing file * tls,http2: send fatal alert on ALPN mismatch nodejs/node#44031 * test: disable snapshot tests * nodejs/node#47887 * nodejs/node#49684 * nodejs/node#44193 * build: use deps/v8 for v8/tools Node.js hard depends on these in their builtins * test: fix edge snapshot stack traces nodejs/node#49659 * build: remove js2c //base dep * build: use electron_js2c_toolchain to build node_js2c * fix: don't create SafeSet outside packageResolve Fixes failure in parallel/test-require-delete-array-iterator: === release test-require-delete-array-iterator === Path: parallel/test-require-delete-array-iterator node:internal/per_context/primordials:426 constructor(i) { super(i); } // eslint-disable-line no-useless-constructor ^ TypeError: object is not iterable (cannot read property Symbol(Symbol.iterator)) at new Set (<anonymous>) at new SafeSet (node:internal/per_context/primordials:426:22) * fix: failing crashReporter tests on Linux These were failing because our change from node::InitializeNodeWithArgs to node::InitializeOncePerProcess meant that we now inadvertently called PlatformInit, which reset signal handling. This meant that our intentional crash function ElectronBindings::Crash no longer worked and the renderer process no longer crashed when process.crash() was called. We don't want to use Node.js' default signal handling in the renderer process, so we disable it by passing kNoDefaultSignalHandling to node::InitializeOncePerProcess. * build: only create cppgc heap on non-32 bit platforms * chore: clean up util:CompileAndCall * src: fix compatility with upcoming V8 12.1 APIs nodejs/node#50709 * fix: use thread_local BuiltinLoader * chore: fixup v8 patch indices --------- Co-authored-by: Keeley Hammond <vertedinde@electronjs.org> Co-authored-by: Samuel Attard <marshallofsound@electronjs.org>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
tools: port js2c.py to C++
This makes it easier to use third-party dependencies in this tool
(e.g. adding compression using algorithms not available in Python).
It is also much faster - locally js2c.py takes ~1.5s to generate the
output whereas this version takes ~0.1s - and consumes less memory
(~110MB v.s. 66MB).
This also modifies the js2c.py a bit to simplify the output, making
it easier to compare with one generated by the C++ version. Locally
the output from the two are identical. We'll remove js2c.py in a
subsequent commit when the C++ version is used by default.
build: replace js2c.py with js2c.cc
Locally, running the two version for the first time (when the output does not yet exist so no reading is done):
And running for the second time (reading the existing output and not overwriting since the content isn't going to change):