[WIP] Persist resolutions and program information and reuse it when building program incrementally or during editor

[sheetalkamat]

This PR adds option persistResolutions which perists the program and resolutions information into the tsbuildinfo file. This option is meant to also mean that if module is resolved to file, dont watch any of the failed lookup locations for those, and always use that file as resolved file unless that file is deleted. The unresolved modules are still resolved again to see if it can be resolved at time of program creation.
This also adds option cleanPersistedProgram to remove the persisted program information (without loosing incremental build information) which helps in creating program as if this option was not enabled. So if user changes location of where the resolution needs to be, they can use this option.

The main changes are:

• Program now takes oldProgram as ProgramFromBuildInfo which is built from program information saved in tsbuildinfo. The stored information is partial information needed to make decisions about whether program structure can be completely reused or only module resolutions can be safely used or cannot reuse at all.
• Missing file paths are calculated on demand instead of saving them in program all the time
• Source files store resolved modules and resolved type reference directives with their failed lookup locations
• During watch, we watch failed lookup locations only if module resolution failed. Successful resolution is not watched.
• Builder change to store the program information needed to make decisions on reuse of structure when persistResolutions is true. Need to be stored with and without --out option.
• Editor reads the tsbuildinfo program and uses that as old program to generate new program so it can also benefit from saved resolutions
• As part of this we also need to make sure that tsserver uses hash of source file text as version as otherwise we wont be able to tell if tsbuildinfo source file is same as version its referring to.
• Rest of the changes are just some refactoring and making sure we are using versions, or partial program/sourceFile info instead of checking if they are strictly equal
• Store if file was from project reference and program and oldProgram differ in useSourceOfProjectReferenceRedirect, then structure cannot be reused and in case of persistentResolution it means module resolution can be reused.
• tsserver logs are generated as baseline to verify details instead of having to check projects programmatically so that its easier to update them

Questions to answer :

1. Is name persistResolutions ok. Given we do need to store more information than just resolutions but resolutions are the main things that determine the behavior of if they should be retained or not.
2. Is cleanPersistedProgram now useful. It was in the start because when i was experimenting , module name resolution whether failed or succeeded would be re-used so if you did npm install, you would want to run this command to clean up. Now that we still resolve failed lookups or if resolved file is deleted, things should flow more smoothly and just deleting tsbuildinfo file should be ok in that rare case.. The new behaviour works better with editing experience.
3. To check how saving resolution when resolved to .js file for eg. is handled when new .d.ts or .ts file is created. (Because without allowJs the resolution will find the js file but wont be part of the program so do we need to do something special for these kind of scenarios) or just let user use the cleanPersistedProgram option. Will it be intuitive?
4. Is cleanPersistedProgram option to be run and then restart the tsserver to handle the the project updates?
5. What kind of reuse we are looking at if we only store module resolution
6. Reuse type reference directive resolution from old program? We just reuse module resolution currently, should it also include type reference resolutions.

[DanielRosenwasser]

This is a really interesting idea. What sorts of speed-ups/memory reduction savings have you seen on larger projects?

[gluxon]

Found this PR again after seeing it on the TypeScript 5.0 roadmap.

This is a really interesting idea. What sorts of speed-ups/memory reduction savings have you seen on larger projects?

I think it'll be extremely significant. Back when I was investigating this for a large monorepo, I saw a 72.21% improvement in one of our team's packages. Details in #40964

Looking forward to some version of this in 5.0. Thanks!