memoized resolve

[tvald]

Summary

Memoize isFile() within default_resolver. Add memoized isDirectory(). Check that a directory exists before attempting to resolve files within that directory.

This provides a huge performance boost to module resolution.

Test plan

[facebook-github-bot]

Thank you for your pull request and welcome to our community. We require contributors to sign our Contributor License Agreement, and we don't seem to have you on file. In order for us to review and merge your code, please sign up at https://code.facebook.com/cla. If you are contributing on behalf of someone else (eg your employer), the individual CLA may not be sufficient and your employer may need the corporate CLA signed.

If you have received this in error or have any questions, please contact us at cla@fb.com. Thanks!

[thymikee]

result !== undefined should perform slightly better, same in isDirectory

[tvald]

Done in 93a7074.

[facebook-github-bot]

Thank you for signing our Contributor License Agreement. We can now accept your code for this (and any) Facebook open source project. Thanks!

[thymikee]

Looks like a leftover?

[tvald]

There are two sections of helper functions - those within the resolveSync() closure, and the two memoized helpers isFile and isDirectory.

[cpojer]

Oh this is super awesome, I'm so glad you are working on this. I'm however afraid that this will not work for things like Jest's watch mode because files may change over time of Jest's usage and the resolution algorithm should work with what is currently on the system, not what used to be there in the past.

To solve this, the Resolver class is able to be instantiated. We can merge the resolution logic into the base class and use a cache within there, and each test will receive it's own copy of the resolver, which will be recreated for individual test runs.

This will not share the resolution logic between two individual test files, so won't be as fast as it could get, but within the dependency graph of a single test, the performance optimization should still apply. We can also change this to memoize for the duration of a single test run, and that would be fine with me, but there is no concept in the worker process that manages the lifetime of a full test run, so we'd have to introduce those hooks somewhere in the workers, which is hard because they are supposed to be stateless. For now, I'd encourage moving all the memoization logic into the Resolver class. Does that make sense to you?

[tvald]

Moving discussion back to #2925, since this will take several stages and multiple PRs. I'm not familiar enough with Jest's codebase to see an easy path beyond the memoization within a single test which you describe. Unfortunately, I don't think that alone will provide enough of a performance boost for Jest to be usable with our codebase.

[niieani]

Hi all!

After doing some profiling, I've independently discovered that memoizing the resolver gives a huge speed-up. Simply adding lodash.memoize to the default_resolver function decreases the run time of our tests by 3x! You've read that right, that's 15 seconds instead of 45 seconds. Granted, our tests have a huge amount of dependencies, so your mileage may vary, but still -- I currently think the resolver is one of the low hanging fruit optimizations that could massively decrease jest's execution time.

[niieani]

All that's needed to make a custom, memoizing resolver based on the default one:

const {default: defaultResolver} = require('jest-resolve/build/default_resolver')
const hasher = require('node-object-hash')({sort: false, coerce: false})
const memoize = require('lodash.memoize')

function hashArgs() { return hasher.hash(arguments) }
module.exports = memoize(defaultResolver, hashArgs)

[sebinsua]

@niieani Where should this code be placed? I would like to try this out. :)

[niieani]

Hi @sebinsua:

1. make a simple package with the above cited file
2. install lodash.memoize, node-object-hash and jest-resolve dependencies to it
3. in your jest config, add "resolver": "path/to/your/resolver"

[lukeapage]

I tried it out on our code-base (730 suites, takes 90 seconds after babel cache is warm). It didn't save any time at all (in fact slightly slower with the cache?) - the resolver was running and it was caching alot of things (I added logging to check) but it seems like a significant amount of time wasn't spent resolving.

[niieani]

@lukeapage that's interesting. It's probably going to vary depending on the codebase. We have a monorepo with lots of projects that are run through jest, all as jest projects, plus a lot of the dependencies are shared. Additionally, our dependency tree is huge (some tests can traverse hundreds of dependencies). Probably that's why the gains on our end are so visible.

[github-actions]

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using StackOverflow or our discord channel for questions.