Improve perf of internal resolve plugin

[patak-dev]

Description

We need to follow Node resolution and also take into account resolve.extensions. Right now a single resolve call could end up exploding into many file system checks.

Creating an umbrella issue to discuss ways to improve the plugin. There are other possibly related issues:

• Vite 2.7.x is about 1.9 to 6.2 times slower than 2.6.14 #6030
• Vite is slow on large codebases on windows #4665

Suggested solution

To be determined.

Alternative

No response

Additional context

No response

Validations

• Follow our Code of Conduct
• Read the Contributing Guidelines.
• Read the docs.
• Check that there isn't already an issue that request the same feature to avoid creating a duplicate.

[sapphi-red]

Some random ideas:

• to reduce JS to C++ overhead caused by calling fs functions many times and stack trace capturing for try-catch
  • we can use some performant internal functions (perf(resolve): improve file existence check #11436 (comment)): tried in perf(resolve): use internalModuleStat #12568
  • parcel uses fs util functions implemented in rust (napi): Implement fs searching in rust parcel-bundler/parcel#5481
• fs.realpath is slower than fs.realpath.native (fs.realpath 70x slower than native  nodejs/node#2680, Different runtime behaviour of fs.realpath vs fs.promises.realpath is very confusing nodejs/node#37737)
  • fs.realpath.native works slight differently and node uses fs.realpath so we cannot simply use fs.realpath.native
  • We currently use fs.realpath.native on non-Windows machines, so if we somehow achieved to make a performant alternative to fs.realpath, we can make Vite faster on Windows.
• negative cache for package.json look up
  • most directories doesn't have package.json so it won't be cached if we only cache hit results
  • Maybe this is already implemented by perf: improve package cache usage #12512 (I haven't checked)
• using package manager's knowledge
  • If package manager's lock files has enough knowledge to identify where the package exists, maybe we can read the lock file and skip file system traversing/real path resolving in some cases?

[bluwy]

• We currently use fs.realpath.native on non-Windows machines, so if we somehow achieved to make a performant alternative to fs.realpath, we can make Vite faster on Windows.

Yeah I think it'll be nice if we can get this fixed for windows. I saw you made a fix before to resolve Windows drive relative paths. It would need some testing to get it work, but I don't have a Windows laptop.

• negative cache for package.json look up

Yeah I think this is implemented in #12512, in a way. Since in most case we will eventually find a package.json somewhere else, the directories traversed to reached there are all cached too so that future lookup is faster.

• using package manager's knowledge

I think it might be good to have Vite be package manager agnostic for now, so future pms like Bun (which saves the lockfile in binary format) will work. Maybe using lockfiles that we understand could help fast-path things, but the idea of JSON.parse or yaml-parsing them is a bit scary 😄

[fi3ework]

using package manager's knowledge

Relying on the knowledge of dependency package managers can be troublesome, and we cannot guarantee that the lockfile is up-to-date. Additionally, Yarn 1 does not guarantee a topological structure. 🤔

[sapphi-red]

I think it might be good to have Vite be package manager agnostic for now ... but the idea of JSON.parse or yaml-parsing them is a bit scary 😄

Relying on the knowledge of dependency package managers can be troublesome, ...

Yeah, I agree with that. I think the ability to skip the FS traverse has great potential but better to be sought as a plugin.

[fi3ework]

Sorry for the misoperation 😅. I also have a speculative idea: is it possible to pre-scan all file structures we need one time and maintained them in memory? Could that reduce the JS/C++ communication overhead to minimal time as we can assume node_modules is immutable. 🤔

[hardfist]

maybe you can try https://github.com/web-infra-dev/nodejs_resolver which we do lots of optimization

[bluwy]

is it possible to pre-scan all file structures we need one time and maintained them in memory? Could that reduce the JS/C++ communication overhead to minimal time as we can assume node_modules is immutable. 🤔

I think that will trade more RAM instead which could be a problem for lower-end machines. We do already cache some crucial files in node_modules through the optimizer, which I think is a step towards it. Caching for source code is already somewhat there with Vite's module graph too.

maybe you can try https://github.com/web-infra-dev/nodejs_resolver which we do lots of optimization

It would be interesting to try that out. Right now Vite mixes a lot optimizer logic within it's node resolver, which could be refactored out.

I'm also concerned of the install size of using native binaries though. IMO it would need a significant improvement to justify the size, that's one of the reason (for me) that @parcel/watcher isn't worth it.

[sapphi-red]

Perhaps this ensureEntryFromUrl call is doubling the resolveId call.

vite/packages/vite/src/node/server/pluginContainer.ts

Line 320 in 1dfecc8

await moduleGraph?.ensureEntryFromUrl(options.id)

vite/packages/vite/src/node/server/moduleGraph.ts

Line 247 in 1dfecc8

const resolved = await this.resolveId(url, !!ssr)

Since we know the id is already resolved, maybe we can skip resolveId? (I didn't find a good way to implement this without introducing a breaking change...)

[patak-dev]

I think we can close this issue for now. Resolving perf in Vite 4.3 has greatly improved. We can continue looking for opportunities to speed up things further but I dont think this umbrella issue is needed now.