-
Notifications
You must be signed in to change notification settings - Fork 30k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
fs: throw errors from sync branches instead of separate implementations #49913
Conversation
Review requested:
|
Previously to throw errors from C++ land, sync versions of the fs were created by copying C++ code from the original implementation and moving JS code to a separate file. This can lead to several problems: 1. By moving code to a new file for the sake of moving, it would be harder to use git blame to trace changes and harder to backport changes to older branches. 2. Scattering the async and sync versions of fs methods in different files makes it harder to keep them in sync and share code in the prologues and epilogues. 3. Having two copies of code doing almost the same thing results in duplication and can be prone to out-of-sync problems when the prologue and epilogue get updated. 4. There is a minor cost to startup in adding an additional file. This can add up even with the help of snapshots. This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4 and introduces C++ helpers SyncCallAndThrowIf() and SyncCallAndThrowOnError() so that the original implementations can be easily tweaked to allow throwing from C++ and stop 3.
9c334ff
to
5dd1e63
Compare
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
My thoughts:
- Moving to sync.js to fs.js seems reasonable and understandable.
- I don't think Fsreqwrap brings any good. I actually believe it brings unnecessary complexity just to avoid a if statement wrapping a trace function.
- Even though, I like your approach I'm extremely frustrated with the inevitable toll this pull request brings to the existing pull requests.
I'm fine with merging this, with the change of the removal of syncfsreq class but I prefer to merge the existing pull requests first to land this.
cc @CanadaHonk
FSReqWrapSync* req_wrap, | ||
Func fn, | ||
Args... args) { | ||
env->PrintSyncTrace(); |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
If the lack of this call causes a bug, I recommend adding a test to address your concerns in the description
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
We do have a test (test/parallel/test-sync-io-option.js
), but it only checks fs.statSync
. Not sure if it's worth it to add a test for every single fs methods, I think it's testing under the assumption that fs methods would share this bit of code so if one is okay, others are fine.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I unknowingly broke it. We should eventually add it.
Sorry that I notice about this too late, but I think at least if we do this soon enough there would be less toll in the future. If we land the existing pull requests some of the problems mentioned in the OP would be difficult to undo (they will create irreversible effect in git history and make Note that the previous approach would also create out-of-sync problems if any of the fs methods being optimized is getting fixed in another PR (obviously we can't block bug fixing just because things are getting optimized), then you need to propogate the fix to the other implementation, and depending on the timing of landing, one of them can be landed without thinking about the other but you have no git conflicts to warn you, which is a bad thing in this case as now the bug lurks again in the other implementation.
I would disagree. The ultimate difference between sync and async version is usually just if/when a callback should be invoked, most of the prologue leading up to the libuv call is and should be shared to avoid getting out of sync - if we can't, at least we can place them close enough to pay attention. It would be easy to accidentally change the prologue of |
I started another benchmar CI run because the one started by @benjamingr had connection issues (queued in https://ci.nodejs.org/view/Node.js%20benchmark/job/benchmark-node-micro-benchmarks/1428/). However I am seeing unexpected performance improvements locally 🤷♀️ Perhaps the benchmarks aren't very stable, or maybe my local fs is too fast so some JS-land factor (e.g. inlining) dominated the numbers.
|
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I'm convinced with the benefits of this pull request, but merging this pull request will block 13 pending pull requests from 6 different contributors. We should merge this once we landed those pull requests to avoid any more frustrations.
Ref: https://github.com/nodejs/node/pulls?q=is%3Aopen+is%3Apr+label%3Aperformance+fs
Fwiw my opinion: I think the approach of this PR is good and better than the existing system. I don't think having open PRs using the "old technique" should block this; PR authors (hopefully) should be able to rewrite easily as they are still relatively small changes (or they can always close and someone else can try). Being a bit more controversial, it would probably aid reviewers by possibly creating one large PR after this and closing existing error perf PRs to use this new technique for ~all sync functions (with pure binding impls/only JS validation and no JS logic) instead of having many small PRs. However, this would make benchmarking results much more difficult being in one PR (and possibly having to create many many benchmark files). Plus, it would block new contributors doing these relatively small but helpful changes via doing one monolithic PR instead (not sure if that would be a concern or not). Regardless, cool work :) |
I think we should do it the other way around - once you land those other fs PRs, the effect they have on git history (for git blame and backports) would be impossible to undo, unless we want to land them all at once, edit all of them and force push, while blocking all other unrelated PRs from landing, but then we might as well just land this first and then ask other PRs to migrate, so that we have a clean git history. |
I agree with Joyee on all points except that I think the splitting of the JS code to a separate sync file is fine, especially if the JS portion of the code is fairly minimal. We already do this with promises, and it makes it easier to only load the parts needed. To me the ideal state would be for the majority of the fs code to all be single C++ functions with helpers to handle sync, callback, and promise forms all in the same function and then have JS files just to expose the different forms and do little else. I do think it's reasonable to hold off on such a split until that unification of the underlying C++ functions is in better shape though. From a perf perspective it might make sense for each to have its own separate C++ function at some point, but only if we can isolate the common parts to helpers that can be reused between each of the forms. Duplicating a bunch of code across different forms is not great. There's a high risk of things getting out-of-sync and that can be a major source of bugs. Performance is important, but not at the risk of stability. It's unfortunate that there's a bunch of PRs in conflict with this, but I agree that it's best to not churn the git history further before landing a cleaner upgrade path. |
Landed in 813713f |
Previously to throw errors from C++ land, sync versions of the fs were created by copying C++ code from the original implementation and moving JS code to a separate file. This can lead to several problems: 1. By moving code to a new file for the sake of moving, it would be harder to use git blame to trace changes and harder to backport changes to older branches. 2. Scattering the async and sync versions of fs methods in different files makes it harder to keep them in sync and share code in the prologues and epilogues. 3. Having two copies of code doing almost the same thing results in duplication and can be prone to out-of-sync problems when the prologue and epilogue get updated. 4. There is a minor cost to startup in adding an additional file. This can add up even with the help of snapshots. This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4 and introduces C++ helpers SyncCallAndThrowIf() and SyncCallAndThrowOnError() so that the original implementations can be easily tweaked to allow throwing from C++ and stop 3. PR-URL: nodejs#49913 Reviewed-By: Stephen Belanger <admin@stephenbelanger.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously to throw errors from C++ land, sync versions of the fs were created by copying C++ code from the original implementation and moving JS code to a separate file. This can lead to several problems: 1. By moving code to a new file for the sake of moving, it would be harder to use git blame to trace changes and harder to backport changes to older branches. 2. Scattering the async and sync versions of fs methods in different files makes it harder to keep them in sync and share code in the prologues and epilogues. 3. Having two copies of code doing almost the same thing results in duplication and can be prone to out-of-sync problems when the prologue and epilogue get updated. 4. There is a minor cost to startup in adding an additional file. This can add up even with the help of snapshots. This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4 and introduces C++ helpers SyncCallAndThrowIf() and SyncCallAndThrowOnError() so that the original implementations can be easily tweaked to allow throwing from C++ and stop 3. PR-URL: nodejs#49913 Reviewed-By: Stephen Belanger <admin@stephenbelanger.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously to throw errors from C++ land, sync versions of the fs were created by copying C++ code from the original implementation and moving JS code to a separate file. This can lead to several problems: 1. By moving code to a new file for the sake of moving, it would be harder to use git blame to trace changes and harder to backport changes to older branches. 2. Scattering the async and sync versions of fs methods in different files makes it harder to keep them in sync and share code in the prologues and epilogues. 3. Having two copies of code doing almost the same thing results in duplication and can be prone to out-of-sync problems when the prologue and epilogue get updated. 4. There is a minor cost to startup in adding an additional file. This can add up even with the help of snapshots. This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4 and introduces C++ helpers SyncCallAndThrowIf() and SyncCallAndThrowOnError() so that the original implementations can be easily tweaked to allow throwing from C++ and stop 3. PR-URL: #49913 Reviewed-By: Stephen Belanger <admin@stephenbelanger.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
* chore: bump node in DEPS to v20.10.0 * chore: update feat_initialize_asar_support.patch no code changes; patch just needed an update due to nearby upstream changes Xref: nodejs/node#49986 * chore: update pass_all_globals_through_require.patch no manual changes; patch applied with fuzz Xref: nodejs/node#49657 * chore: update refactor_allow_embedder_overriding_of_internal_fs_calls Xref: nodejs/node#49912 no code changes; patch just needed an update due to nearby upstream changes * chore: update chore_allow_the_node_entrypoint_to_be_a_builtin_module.patch Xref: nodejs/node#49986 minor manual changes needed to sync with upstream change * update fix_expose_the_built-in_electron_module_via_the_esm_loader.patch Xref: nodejs/node#50096 Xref: nodejs/node#50314 in lib/internal/modules/esm/load.js, update the code that checks for `format === 'electron'`. I'd like 👀 on this Xref: nodejs/node#49657 add braces in lib/internal/modules/esm/translators.js to sync with upstream * fix: lazyload fs in esm loaders to apply asar patches * nodejs/node#50127 * nodejs/node#50096 * esm: jsdoc for modules code nodejs/node#49523 * test: set test-cli-node-options as flaky nodejs/node#50296 * deps: update c-ares to 1.20.1 nodejs/node#50082 * esm: bypass CommonJS loader under --default-type=module nodejs/node#49986 * deps: update uvwasi to 0.0.19 nodejs/node#49908 * lib,test: do not hardcode Buffer.kMaxLength nodejs/node#49876 * crypto: account for disabled SharedArrayBuffer nodejs/node#50034 * test: fix edge snapshot stack traces nodejs/node#49659 * src: generate snapshot with --predictable nodejs/node#48749 * chore: fixup patch indices * fs: throw errors from sync branches instead of separate implementations nodejs/node#49913 * crypto: ensure valid point on elliptic curve in SubtleCrypto.importKey nodejs/node#50234 * esm: detect ESM syntax in ambiguous JavaScrip nodejs/node#50096 * fixup! test: fix edge snapshot stack traces * esm: unflag extensionless ES module JavaScript and Wasm in module scope nodejs/node#49974 * [tagged-ptr] Arrowify objects https://chromium-review.googlesource.com/c/v8/v8/+/4705331 --------- Co-authored-by: electron-roller[bot] <84116207+electron-roller[bot]@users.noreply.github.com> Co-authored-by: Charles Kerr <charles@charleskerr.com> Co-authored-by: Shelley Vohr <shelley.vohr@gmail.com>
Previously to throw errors from C++ land, sync versions of the fs were created by copying C++ code from the original implementation and moving JS code to a separate file. This can lead to several problems: 1. By moving code to a new file for the sake of moving, it would be harder to use git blame to trace changes and harder to backport changes to older branches. 2. Scattering the async and sync versions of fs methods in different files makes it harder to keep them in sync and share code in the prologues and epilogues. 3. Having two copies of code doing almost the same thing results in duplication and can be prone to out-of-sync problems when the prologue and epilogue get updated. 4. There is a minor cost to startup in adding an additional file. This can add up even with the help of snapshots. This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4 and introduces C++ helpers SyncCallAndThrowIf() and SyncCallAndThrowOnError() so that the original implementations can be easily tweaked to allow throwing from C++ and stop 3. PR-URL: nodejs#49913 Reviewed-By: Stephen Belanger <admin@stephenbelanger.com> Reviewed-By: Colin Ihrig <cjihrig@gmail.com> Reviewed-By: Darshan Sen <raisinten@gmail.com> Reviewed-By: Benjamin Gruenbaum <benjamingr@gmail.com> Reviewed-By: Tobias Nießen <tniessen@tnie.de> Reviewed-By: Yagiz Nizipli <yagiz@nizipli.com>
Previously to throw errors from C++ land, sync versions of the fs were created by copying C++ code from the original implementation and moving JS code to a separate file. This can lead to several problems:
This patch moves the JS code back to lib/fs.js to stop 1, 2 & 4 and introduces C++ helpers SyncCallAndThrowIf() and SyncCallAndThrowOnError() so that the original implementations can be easily tweaked to allow throwing from C++ and stop 3.