-
Notifications
You must be signed in to change notification settings - Fork 29.6k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Reduce syscalls on require() #1920
Reduce syscalls on require() #1920
Conversation
@@ -142,6 +142,8 @@ Module._findPath = function(request, paths) { | |||
|
|||
// For each path | |||
for (var i = 0, PL = paths.length; i < PL; i++) { | |||
// don't search further if path doesnt exist | |||
if(paths[i] && internalModuleStat(paths[i]) <= 0) continue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
There should at least be a space here between if
and (
.
The comment should be modified slightly to read: // Don't search further if path doesn't exist
The subsystem the commit is targeting should be |
951fc5b
to
a2d873c
Compare
Is this before or after #1801, or does that not make a difference? EDIT: ah that may not matter. |
It's after but it makes no difference. |
If this can improve node startup time, I'm all for it. Some of my CLIs take 400-500ms to start up, all in require, mostly statting non-existent directories. That doesn't sound like much, but under unit test, when the CLIs are spawned dozen's of times per test, it adds up very quickly. |
Change looks sound. @isaacs Know of a reason why this would be a problem? |
Ah, didn't realize this was a PR. SGTM.. CI: https://jenkins-iojs.nodesource.com/view/iojs/job/iojs+any-pr+multi/787/ |
Looking at the CI I do not think this patch works on FreeBSD, SmartOS, or Windows. |
The errors are suite strange, both freebsd throw 'Error: listen EADDRINUSE 0.0.0.0:12346' |
It does appear that everything that failed is network related. @rvagg You seen this happen before with Jenkins? |
I found lingering processes on some machines (freebsd, smartos). We just discussed this in the build group and will be looking at making these tests more robust and/or reaping processes on test exit. |
@jbergstroem should we be good to run CI again? |
@trevnorris feel free. The windows bots at least seems to give proper feedback (800+ failing tests because they can't be run): https://jenkins-iojs.nodesource.com/view/iojs/job/iojs+any-pr+multi/787/nodes=win2008r2/tapTestReport/test.tap-5/ |
@jbergstroem thanks. looks like jenkins hasn't been happy the last few builds. :( Having another go at it: https://jenkins-iojs.nodesource.com/job/iojs+any-pr+multi/789/ |
@trevnorris yeah, its really unfortunate. |
The change seems fine to me. However, I'd love to see a test for this. Is it possible to monkey-patch fs.statSync and track which things get statted to create a failing test? |
Oh, I guess with the change in #1801, testing this might be a bit tricky. @bnoordhuis do you have any insight? If it's just not possible, and it doesn't make any existing tests fail (and doesn't, eg, break npm's or request's or express's tests) then LGTM. |
this does make windows blow up hard:
I'm guessing this error is specifically related to this PR. |
I wonder how many packages test for presence of a module, generate it on "does not exist", then re-require it? I wouldn't advocate such an approach, of course, but it's within the realm of possibility. Do we want to support that use case? |
@chrisdickinson how would that even be implemented? |
try {
var ourCompiledModule = require('./generated.js')
} catch (err) {
fs.writeFileSync(__dirname + '/generated.js', 'module.exports = 3', 'utf8');
} Also, consider the REPL use case:
|
@chrisdickinson do you mention that as a possible new feature? Or is that something that works today? |
During require search, check the path exists before searching further in it.
a2d873c
to
a10ce67
Compare
On Unix systems, when the request path was absolute, it removed all the search paths but not on Windows, the |
@trevnorris as a thing that works today that would not work if we cached "ENOENT" results to avoid syscalls. |
This PR doesn't cache 'ENOENT' between require() calls, it just avoid calling stat/open on files if the directory doesn't exist. That wont break your usecase. |
@trevnorris Could you run this PR on jenkins again ? |
@bnoordhuis thanks. CI looks happy. LGTM. |
One second – going to check this locally first just to make sure / prove that I'm just being paranoid. |
Myth: CONFIRMED. I was just being paranoid. LGTM :) |
require() now checks that the path exists before searching further in it. PR-URL: #1920 Reviewed-By: Isaac Z. Schlueter <i@izs.me> Reviewed-By: Trevor Norris <trev.norris@gmail.com> Reviewed-By: Chris Dickinson <christopher.s.dickinson@gmail.com> Reviewed-By: Jeremiah Senkpiel <fishrock123@rocketmail.com>
Thanks, landed in a71ee93! |
@@ -142,6 +142,8 @@ Module._findPath = function(request, paths) { | |||
|
|||
// For each path | |||
for (var i = 0, PL = paths.length; i < PL; i++) { | |||
// Don't search further if path doesn't exist | |||
if (paths[i] && internalModuleStat(paths[i]) < 1) continue; |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I guess that one needs the path._makeLong
fix (#1991) ?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Oops, yes, I think so. PR: #2011
PR-URL: #1996 Notable changes * module: The number of syscalls made during a require() have been significantly reduced again (see #1801 from v2.2.0 for previous work), which should lead to a performance improvement (Pierre Inglebert) #1920. * npm: - Upgrade to v2.11.2 (Rebecca Turner) #1956. - Upgrade to v2.11.3 (Forrest L Norvell) #2018. * zlib: A bug was discovered where the process would abort if the final part of a zlib decompression results in a buffer that would exceed the maximum length of 0x3fffffff bytes (~1GiB). This was likely to only occur during buffered decompression (rather than streaming). This is now fixed and will instead result in a thrown RangeError (Michaël Zasso) #1811.
PR-URL: #1996 Notable changes * module: The number of syscalls made during a require() have been significantly reduced again (see #1801 from v2.2.0 for previous work), which should lead to a performance improvement (Pierre Inglebert) #1920. * npm: - Upgrade to v2.11.2 (Rebecca Turner) #1956. - Upgrade to v2.11.3 (Forrest L Norvell) #2018. * zlib: A bug was discovered where the process would abort if the final part of a zlib decompression results in a buffer that would exceed the maximum length of 0x3fffffff bytes (~1GiB). This was likely to only occur during buffered decompression (rather than streaming). This is now fixed and will instead result in a thrown RangeError (Michaël Zasso) #1811.
In some conditions, require makes many useless syscalls trying to find files in non-existent directories.
For example, this is a
require('moment');
in one subfolder of my project, the lib/sub/node_modules doesn't even exist.This PR divides by 7 the number of syscalls when requiring a module from a directory without node_modules at the cost of adding 1 from a directory with.
From my tests with a whole process doing only
require('express')
, I pass from 760stat64
to 574 (-25%) and from 106open_nocancel
to 58 (45%).