TypeScript Language Service is slow to load projects over network file systems

[vikerman]

TypeScript Version: 2.3.4

We have our code on a networked/FUSE file system with different root directories for generated files and such. We use the node module resolution. To load a fairly simple Angular+Angular material+RxJS project(transitively ~1.7K .ts and .d.ts files) the language service takes around 24 seconds to resolve all modules and return diagnostics. We can see about ~50K file stats being made to resolve everything.

In our case we actually generate tsconfig.json for our editors through a build step and we actually create the list of all the files the project needs in the "files" section of tsconfig.json.

We built a custom solution that uses this files list to proxy the ServerHost in the Language service - to respond to fileExists and directoryExists by using the "files" list instead of hitting the file system a whole lot of times. This reduced the initial project load time to 4 seconds (and is mostly just the overhead of reading all the files) which is very much an

Here is a PR of the fix we have currently - https://github.com/vikerman/TypeScript/pull/1/files

It would be nice to have a cleaner solution for this in the language service itself (maybe as an option?). This would help any team who have a similar network/FUSE file system as their source repository.

@chuckjaz @alexeagle

[DanielRosenwasser]

So the idea is to always assume a file exists if it's explicitly listed? IIRC this used to be the old behavior but we stopped that because it was actually slower with Node's module resolution on local file systems.

[vikerman]

So the underlying problem is that number of fstat-s in the node module resolution is too high and it is very visible on a network drive. Our current solution is by overriding file/directoryExists - but maybe you can suggest some other way to solve it also.

And we need a clean way to integrate with the language service so that we don't have to maintain a separate entrypoint/binary. The LS plugin mechanism (#11976) was insufficient because the base LSHost is already created with the sys ServerHost and there is no way to override it.

[mhegazy]

Do you need the node module resolution? can you share a sample --traceResolution output, i would like to understand what modules are we looking for and where.

[vikerman]

We need node module resolution because we use path mapping. Related - #11979

Will get the logs for the traceResolution shortly.

[mhegazy]

path mapping is not tied to node module resolution.

[DanielRosenwasser]

@mhegazy the problem, IIRC, is that if you use "classic", you also get inappropriate file-statting in a different way when walking up the directory spine. I think a mix of moduleResolution: "none" and moduleResolution: "node" is needed, but @vikerman and the team can correct me if I'm wrong.

Basically:

1. Loading from package.json and the like is potentially needed from path-mapped directories.
2. Walking up each directory's node_modules is not needed.

[vikerman]

1. We actually don't need package.json logic also. We do depend on some minimal parts of the node module lookup like looking for /index (Not sure if classic has that one)
2. Walking up for node_modules and @types are definitely not needed.

[mhegazy]

Looking through the trace from @vikerman, we are doing too many lookups for imports in .d.ts file, e.g. rxjs importing other parts of rxjs. we should optimize that. that will not remove all lookups but should reduce them by a big factor.

If that does not work either, we can try to add a flag to limit file lookups, but i would rather do that as a last resort.

[DanielRosenwasser]

I think we're going to have to push this to the 2.6 milestone until we can discuss any solutions with @mhegazy when he comes back.

[DanielRosenwasser]

Just to keep the two linked together, there's also an issue tracking pluggable module resolution at #18896.