Build improvements

[sheetalkamat]

Commits: 16cef4a, 59ad6ef, c8327da

• During tsc --build and other scenarios where we would do existence check as well as stat which resulted in two stat calls, we just check modified time

Commits: cb7aca3

• In tsc --build while determining upto date ness status, we now check if referenced projects have errors so that we dont need to go through input and output timestamps in that case

Commits: 89d2d4c

• During watch especially during polling file watch and in some scenarios during native file-system based watch we stat on files and that time is now passed through as part of file watcher event so tsc --build can use it.

Commits: 6e0c916

• Apart from this in watch mode we save timestamps for all the watched files. These get updated through watcher event callback or are repopulated if watcher event didn't query the time. So if 2 files changed out of 20 input files then at max we will requery timestamps for those two files (if watch callback didn't already give us updated time)
  Fixes Build mode explicitly checks file stamps in watch mode rather than using file watcher events #45082

Commits: 7cb0f40, 1a8abac, 59f2b5c, dc21283, 5bccee8,

• If its incremental build, tsbuildInfo alone determines upto date ness of the build. For this we now store changeFileSet, dtsChangeTime and emitSignatures for the output files in the buildInfo when in multi file emit scenario.
• We store fileVersions, outSignature, dtsChangeTime in single file output scenario.
• The emitSignatures and outSignature are the d.ts hash of the output that last changed to the disk. Since any composite project already has declaration as must we can store this easily and save having to read files on emit to see if the d.ts file has changed. This makes sure that whether you are in tsc --build or tsc --build --watch mode the state is persisted so we know exactly if we need to really build or just update timestamps.
• In case of multi file emit scenario, tsbuildinfo timestamp, changeFileSet and errors in the buildInfo determine whether we need to rebuild project or not. This way we dont have to update timestamps for anything except buildInfo when thing appear uptodate but timestamp doesnt reflect that. (eg upsteam project changed but didnt change d.ts files)
• In case of single file emit scenario, tsbuildInfo timestamp is enough because we cant emit tsbuildinfo without emitting other output files.
• Earlier we checked input file timestamps and found newest file, now in case of incremental build, we first check buildInfo as missing it, or errors is clear indication of requiring rebuild. After that we check each input file timestamp against buildInfo so that the number of checks can be smaller.
• Fixes Experiment with storing output timestamps in .tsbuildinfo #46677
• Fixes tsc --build --watch (with or without --incremental) touches all files causing problems for downstream tooling #46661

Commits: 2f2e370, fcf07f8, 15fe24e

• In non incremental project, we still need to update timestamps for output files that didnt change as there is no other way to tell if project is upto date or not. But we dont need to read d.ts files to see if .d.ts files have chaned since these projects cannot be referenced by other projects. (composite also means incremental which is must for referenced proejcts)

Commits: 6198fa3,

• With so much depending on buildInfo, it is now cached and passed through emit as additional data so we dont have to reparse the json or identify which file is tsbuildinfo

Commits: 62c687b

• We store the hash of files written in bundle emit to ensure that when we try to manipulate the sources for pseudo update (because d.ts of referenced project didnt change so only prepends need to concatenated in right order), we can check if file hasnt changed. This is now needed because bundle buildInfo also stores .d.ts change time and if incremental was toggled between compilations we might use wrong file text and get incorrect d.ts Look at 62c687b for details and 62c687b#diff-6b83fa9d09e38f88c61b24a70aec8668c68e0d8322e8404dbd85c90120f14eeeL1016 shows test that failed without this change

Commits: 4fb6773

• We also cache output file timestamps so that we dont have to requery as we are the one who write those files.

Commits: 0f7903d, 437619e

• This also has some refactoring of BuilderState, BuilderProgramState etc structures so its correctly reflected.
• With this change we also have flags for emitting compiler option key into the buildInfo based on multi file or single file emit

Commits: e6a3ee8

• tsc --build --force now doesnt query input file stamps (it never did query output file stamps)

Commits: b32d2eb

• In builder, earlier we use to copy too many structures as part of backup which is internal method to ensure we can restore state if declaration emit fails. So now we are only storing emit state instead of reference map etc

Commits: 303824e

• Other change in builder is that earlier we use to maintain new delta of changes for affected files new signatures as well as new exported modules map. We use to commit it once all the affected files for the changed file is handled. (This was due to our API usage where we could cancel the operations in between and would still need to be able to reuse the state as much as possible). Now i have changed this to instead maintain copy of old signatures and old export map so that all we do is clear these these maps instead of having iterate through each and update them into the latest copy on commit

Commits: 28a9ff3

• During tsc --build if file timestamp is changed but its text has not, we read the file and do pseudo update (only in incremental build).
• Fixes --watch compilation with no change takes 3 seconds #31932

Commits: e4e6672 - Delete only tsbuildinfo instead of all output files when doing clean on incremental project

Here are the numbers from run on solution with large (1994) number of projects.

	main	PR	Delta	%	Projects Built	Timestamps Updated	Total projects
tsc -b	120.21	93.38	-26.83	-22.32%	255	0	1994
tsc -b (Fix one error)	39.13	31.50	-7.63	-19.50%	22	0	1994
tsc -b (Fix multiple errors in files)	43.96	35.74	-8.22	-18.70%	43	0	1994
tsc -b -w	373.25	326.22	-47.03	-12.60%	484	0	1994
Fix error	19.69	17.76	-1.93	-9.80%	11	0	1994
Fix error	61.48	60.71	-0.77	-1.25%	66	0	1994
Fix error	11.53	8.97	-2.56	-22.20%	3	0	1994
Fix error	7.19	6.18	-1.01	-14.05%	2	0	1994
tsc -b	620.07	480.23	-139.84	-22.55%	796	0	1994
tsc -b No change	26.78	19.59	-7.20	-26.87%	6	0	1994
Local Change	36.70	20.14	-16.57	-45.13%	7	477	1994
D.ts change Average	74.60	54.47	-20.12	-26.98%	138	346	1994
tsc -b -w	630.56	557.41	-73.15	-11.60%	796	0	1994
Local Change	8.98	7.52	-1.46	-16.21%	5	131	1994
D.ts change Average	89.21	70.56	-18.65	-20.90%	136	133	1994
tsc -b -w no change	37.58	26.37	-11.21	-29.83%	6	0	1994

On Typescript Code base:

Total Projects In the	main	PR	Delta	%	Projects Built	Timestamps Updated	Bundles updated	Total Project
tsc -b	60.73	49.03	-11.70	-19.26%	18	0	0	19
tsc -b no change	0.15	0.09	-0.06	-41.56%	0	0	0	19
tsc -b local change	33.42	26.17	-7.25	-21.68%	1	4	9	19
tsc -b dts change	58.87	45.63	-13.24	-22.48%	14	0	0	19

[sheetalkamat]

@amcasey @andrewbranch i have highlighted important commits and the change in the description and also have left some comments during the change. Let me know if you need more details.
This PR is basically aggregation of lot of changes that mostly needs to be together.. So minor refactoring can be a separate PR but rest needs to be together for everything to work so cant make this into smaller PR.

[amcasey]

@sheetalkamat That table's great! When you ran those scenarios, did you do any checks to determine whether the output was the same as before? Also, do you have a similar (or simpler) table for compiling TypeScript itself?

[amcasey]

Your offline explanation made sense and I appreciate your thorough perf testing. I added some notes on naming and comments, but mostly I think we should check this in ASAP and see what happens.

[NickHeiner]

I think this might have caused a regression. In my project, when I introduce errors and build with tsc -p, I get errors, as expected. When I build with tsc --build, it exits very quickly, and shows no errors.

λ ./tools/tvui/node_modules/.bin/tsc -p packages/tsconfig.json 
# ... expected error messages omitted

Found 3 errors in 3 files.

Errors  Files
     1  node_modules/relay-hooks/lib/RelayHooksTypes.d.ts:56
     1  packages/@tvui/fx2-transition/src/utils.tsx:173
     1  packages/@tvui/redux/testing/unitTestingEnhancer.ts:8

tvui on  nth/ts-4.8 [!⇡] via  v16.16.0 took 10s 

λ time ./tools/tvui/node_modules/.bin/tsc --build packages/tsconfig.json --verbose
[3:52:58 PM] Projects in this build: 
    * packages/tsconfig.json

[3:52:58 PM] Project 'packages/tsconfig.json' is up to date but needs to update timestamps of output files that are older than input files


________________________________________________________
Executed in  550.41 millis    fish           external
   usr time  437.83 millis   50.00 micros  437.78 millis
   sys time  194.10 millis  616.00 micros  193.49 millis

--build does seem to be faster, but perhaps it's so fast that it's not doing any work. 😄

I noticed this when doing the TS 4.8 upgrade. I observe this with version 4.8.2.

I'm happy to open a new issue for this if that's preferable.

I first noticed this after using yarn to update type-fest in node_modules.

[pp0rtal]

@NickHeiner

I'm happy to open a new issue for this if that's preferable.

New issue is definitively better, check if there is no similar issue.
If you have a standalone minimal project to reproduce it will be gold for maintainers 🤗
Thank you!