Skip to content
This repository has been archived by the owner on Apr 22, 2023. It is now read-only.

module: fix addition of filetypes, if require ends with .. #7094

Closed

Conversation

robertkowalski
Copy link

Given my home-directory is /Users/rocko - and I have a file named
npm.json in it and also a repository with name npm, which is a
node-module.

When try to require the /Users/rocko/npm/index.js two direcotry
levels down in the npm folder (e.g. /Users/rocko/npm/test/tap)
with require("../../") node will load /Users/rocko/npm/index.js.

When I use require("../..") node will load /Users/rocko/npm.json
which is fixed by this commit.

@Nodejs-Jenkins
Copy link

Thank you for contributing this pull request! Here are a few pointers to make sure your submission will be considered for inclusion.

Commit robertkowalski/node@b382ced has the following error(s):

  • First line of commit message must be no longer than 50 characters

The following commiters were not found in the CLA:

  • Robert Kowalski

You can fix all these things without opening another issue.

Please see CONTRIBUTING.md for more information

@robertkowalski
Copy link
Author

You lied to me, Jenkins!

Please visit http://nodejs.org/cla.html and sign the Contributor License Agreement. 
You only need to do that once.

Signed the CLA again (last commit is almost a year old)

@trevnorris
Copy link

Jenkins is broken. Don't worry about it. :-)

@robertkowalski
Copy link
Author

it's been a few months, any chance that this gets into a release?

@trevnorris
Copy link

@tjfontaine Does this functionality better match what you'd expect?

@tjfontaine
Copy link

I spoke some with @robertkowalski in the irc channel, there are a lot of things in the module loader that behave implicitly, for instance relying on the ordering of property iteration from v8 as opposed to explicitly defining the preferred ordering. It's not always easy to fix things in the module loader without also breaking the legacy tail.

In this particular case the intention is to tryPackage when what we have is indeed a directory. Special casing ../.. is not particularly satisfying when instead we could perhaps be doing a more crisp check asking if the path we're considering is truly a directory as compared to checking if it ends in a slash or if we have a filename. Granted those decisions were made to keep the module loader fast to prevent doing a lot of stat()ing but perhaps there's a saner way to iterate this.

It's important that how this works also falls in line with how the work we want to do for bundles is done, so I'm going to /cc @bmeck

@bmeck
Copy link
Member

bmeck commented Jun 14, 2014

The logic should not affect bundles. Bundles are treated as files that when encountered act like directories. Though, I do think the extra stat calls are going to hit even when we do use bundles since you need to associate directories w/ which kind of loader the directory has (bundled vs. fs).

@robertkowalski
Copy link
Author

Ok, had some offline conversation with TJ in Lisbon.

Here is what I did:

I updated the PR to use statSync

Created some benchmarks, which can be found at: https://github.com/robertkowalski/require-benchmark - I first created the data, then rebooted and started testing. After each run I rebooted again. After each reboot I waited 20 sec.

I think it is important to note that module.js uses the statSync call already in another place.

The first version of my benchmark had a little error that led to interesting end results:
it seems if I miss the package in tryPackage and node gets into tryExtensions the benchmarks are slower with the added isDirectory call.

If a package with tryPackage (the test i wanted to test first but did not in the first place) is found, the modified version is slightly faster than the one one master. The reason for this are probably calls to isDirectory and stat for three times in the tryExtensions function which is not called if I already found my directory.

So we'll see that the modified version, which tests for directories is slightly faster in the synthetic testcases nested and flat. I think this happens as the code path which tries to append several extensions is not used in this version of the benchmark any more (before the benchmark just tested the addition of the isDirectory call and the codepath ended in the last method which tested if a file with some extensions exist. tryExtensions calls tryFile which also has a isDirectory call 3 times(see a diff of both benchmark versions robertkowalski/require-benchmark@0cc5fde)

As mentioned, it gets slower (around 1000s on the first call) if the isDirectory is called, but additionally the code has to test for the extensions and the variable filename does not get truthy (happens in the old version of the benchmark: https://github.com/robertkowalski/require-benchmark/blob/5e4d23782c5ccb17b1afc692303017b197607fe7/create-data.sh)

Results require-benchmark@latest (0cc5fde0ee83524bea97d4bd4f6296f981aa4a99) :

hapi - modified

(01:02:41) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark hapi
node: 2133ms
node: 836ms
node: 849ms
node: 882ms
node: 840ms
node: 825ms
node: 868ms
node: 828ms
node: 781ms
node: 784ms
node: 812ms

Total time:   10438 ms
Average time: 1043 ms

hapi - current master

(01:15:52) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark hapi
node: 1798ms
node: 830ms
node: 831ms
node: 825ms
node: 847ms
node: 867ms
node: 870ms
node: 859ms
node: 822ms
node: 825ms
node: 839ms

Total time:   10213 ms
Average time: 1021 ms

flat - modified

(01:55:42) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark flat
node: 12618ms
node: 3847ms
node: 4049ms
node: 3967ms
node: 3948ms
node: 3855ms
node: 3939ms
node: 3855ms
node: 3924ms
node: 3846ms
node: 3956ms

Total time:   51804 ms
Average time: 5180 ms

flat - current master

(02:02:48) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark flat
node: 13916ms
node: 5250ms
node: 5108ms
node: 4993ms
node: 5063ms
node: 5025ms
node: 5056ms
node: 5007ms
node: 5058ms
node: 4969ms
node: 5013ms

Total time:   64458 ms
Average time: 6445 ms

nested - modified

(02:00:40) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark nested
node: 2ms
node: 2ms
node: 1ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 1ms
node: 2ms
node: 2ms

Total time:   20 ms
Average time: 2 ms

nested - current master

(02:06:27) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark nested
node: 3ms
node: 2ms
node: 2ms
node: 2ms
node: 3ms
node: 3ms
node: 3ms
node: 2ms
node: 2ms
node: 2ms
node: 3ms

Total time:   27 ms
Average time: 2 ms

old benchmark versions (the ones that use tryExtensions as they miss a package)

flat - modified - old version of benchmark

(01:06:35) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark flat
node: 8032ms
node: 2945ms
node: 2920ms
node: 3012ms
node: 3091ms
node: 2884ms
node: 2916ms
node: 2946ms
node: 2912ms
node: 2888ms
node: 2943ms

Total time:   37489 ms
Average time: 3748 ms

flat - current master - old version of benchmark

(01:18:04) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark flat
node: 9030ms
node: 2365ms
node: 2351ms
node: 2278ms
node: 2271ms
node: 2353ms
node: 2269ms
node: 2268ms
node: 2305ms
node: 2245ms
node: 2257ms

Total time:   31992 ms
Average time: 3199 ms

nested - modified - old version of benchmark

(01:11:54) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark nested
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms

Total time:   22 ms
Average time: 2 ms

nested - current master - old version of benchmark

(01:20:19) [robert@tequila-osx] ~/require-benchmark (master *) $ ./run-benchmark nested
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 2ms
node: 1ms
node: 2ms
node: 2ms
node: 2ms
node: 1ms

Total time:   20 ms
Average time: 2 ms

@robertkowalski robertkowalski force-pushed the fix/require-slice branch 2 times, most recently from f0580d5 to 72f26cc Compare September 9, 2014 13:48
@robertkowalski
Copy link
Author

@tjfontaine let's talk during this nodeconf.eu about that

Given my home-directory is `/Users/rocko` - and I have a file named
`npm.json` in it and also a repository with name `npm`, which is a
folder for the node-module.

When try to require the `/Users/rocko/npm/index.js` two direcotry
levels down in the npm folder (e.g. `/Users/rocko/npm/test/tap`)
with require("../../") node will load `/Users/rocko/npm/index.json`.

When I use require("../..") node will load `/Users/rocko/npm.json`
which is fixed by this commit.
@robertkowalski
Copy link
Author

Using the new included benchmark I get the following results:

master

(17:44:18) [robert@tequila-osx] ~/projects/node (master) $ out/Release/node benchmark/misc/module-loader.js
misc/module-loader.js thousands=50: 1.7559
(17:45:58) [robert@tequila-osx] ~/projects/node (master) $ out/Release/node benchmark/misc/module-loader.js
misc/module-loader.js thousands=50: 1.7554

0.12

(17:50:12) [robert@tequila-osx] ~/projects/node (v0.12) $ out/Release/node benchmark/misc/module-loader.js
misc/module-loader.js thousands=50: 1.9697
(17:51:55) [robert@tequila-osx] ~/projects/node (v0.12) $ out/Release/node benchmark/misc/module-loader.js
misc/module-loader.js thousands=50: 1.9939

patch

(18:02:36) [robert@tequila-osx] ~/projects/node (fix/require-slice) $ out/Release/node benchmark/misc/module-loader.js
misc/module-loader.js thousands=50: 1.8188
(18:04:26) [robert@tequila-osx] ~/projects/node (fix/require-slice) $ out/Release/node benchmark/misc/module-loader.js
misc/module-loader.js thousands=50: 2.3875

@jasnell
Copy link
Member

jasnell commented Aug 3, 2015

@robertkowalski ... still need/want this?

@robertkowalski
Copy link
Author

it landed in io.js along with other improvements to the module loader

On Mon, Aug 3, 2015 at 11:10 PM, James M Snell notifications@github.com
wrote:

@robertkowalski https://github.com/robertkowalski ... still need/want
this?


Reply to this email directly or view it on GitHub
#7094 (comment).

@jasnell
Copy link
Member

jasnell commented Aug 4, 2015

Do you see a reason to land it in v0.12? Or can we close this and pick it up under the converged stream?

@robertkowalski
Copy link
Author

it is a bugfix, but a breaking fix. the idea was to land it in 0.11 and get it into the 0.12 release. i am happy if it lands in the next versions, so we can close it. :)

@robertkowalski robertkowalski deleted the fix/require-slice branch August 4, 2015 16:19
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging this pull request may close these issues.

6 participants