Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[experiment] services monomorphization + tsc using public API #58080

Closed
wants to merge 5 commits into from
Closed

[experiment] services monomorphization + tsc using public API #58080

wants to merge 5 commits into from

Conversation

jakebailey
Copy link
Member

Temp PR to perf test #58045 + #57703, i.e. "how much perf difference is there between tsc versus tsc+services allocators?"

@typescript-bot typescript-bot added Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug labels Apr 4, 2024
@jakebailey
Copy link
Member Author

@typescript-bot perf test this

@typescript-bot
Copy link
Collaborator

typescript-bot commented Apr 4, 2024 •
edited
Loading

Starting jobs; this comment will be updated as builds start and complete.

Command Status Results
perf test this ✅ Started 👀 Results

@typescript-bot
Copy link
Collaborator

Looks like you're introducing a change to the public API surface area. If this includes breaking changes, please document them on our wiki's API Breaking Changes page.

Also, please make sure @DanielRosenwasser and @RyanCavanaugh are aware of the changes, just as a heads up.

@typescript-bot
Copy link
Collaborator

@jakebailey
The results of the perf run you requested are in!

Here they are:

tsc

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Angular - node (v18.15.0, x64)
Memory used 295,667k (± 0.01%) 301,910k (± 0.01%) +6,243k (+ 2.11%) 301,867k 301,945k p=0.005 n=6
Parse Time 3.93s (± 0.54%) 4.04s (± 0.30%) +0.11s (+ 2.76%) 4.03s 4.06s p=0.004 n=6
Bind Time 1.22s (± 0.68%) 1.25s (± 1.10%) +0.03s (+ 2.45%) 1.24s 1.27s p=0.007 n=6
Check Time 12.04s (± 0.25%) 12.15s (± 0.34%) +0.11s (+ 0.90%) 12.11s 12.20s p=0.005 n=6
Emit Time 10.48s (± 0.32%) 10.51s (± 0.20%) ~ 10.48s 10.54s p=0.141 n=6
Total Time 27.68s (± 0.17%) 27.95s (± 0.22%) +0.28s (+ 1.01%) 27.86s 28.03s p=0.005 n=6
Compiler-Unions - node (v18.15.0, x64)
Memory used 193,345k (± 0.95%) 198,182k (± 0.23%) +4,837k (+ 2.50%) 197,538k 198,552k p=0.005 n=6
Parse Time 2.02s (± 1.71%) 2.08s (± 1.24%) +0.06s (+ 3.22%) 2.04s 2.12s p=0.012 n=6
Bind Time 1.08s (± 1.28%) 1.09s (± 0.75%) ~ 1.08s 1.10s p=0.183 n=6
Check Time 13.94s (± 0.51%) 14.01s (± 0.62%) ~ 13.92s 14.13s p=0.261 n=6
Emit Time 3.85s (± 0.55%) 3.88s (± 4.36%) ~ 3.74s 4.11s p=0.375 n=6
Total Time 20.88s (± 0.34%) 21.06s (± 0.77%) ~ 20.85s 21.25s p=0.054 n=6
Monaco - node (v18.15.0, x64)
Memory used 347,524k (± 0.00%) 353,488k (± 0.00%) +5,964k (+ 1.72%) 353,465k 353,500k p=0.005 n=6
Parse Time 2.99s (± 0.99%) 2.94s (± 3.87%) ~ 2.82s 3.09s p=0.421 n=6
Bind Time 1.11s (± 0.46%) 1.23s (± 8.29%) ~ 1.10s 1.31s p=0.367 n=6
Check Time 8.27s (± 0.55%) 8.25s (± 0.20%) ~ 8.23s 8.27s p=0.376 n=6
Emit Time 4.84s (± 0.34%) 4.90s (± 0.44%) +0.06s (+ 1.14%) 4.87s 4.93s p=0.006 n=6
Total Time 17.21s (± 0.36%) 17.32s (± 0.19%) +0.11s (+ 0.63%) 17.29s 17.37s p=0.012 n=6
TFS - node (v18.15.0, x64)
Memory used 302,835k (± 0.01%) 308,538k (± 0.01%) +5,703k (+ 1.88%) 308,514k 308,570k p=0.005 n=6
Parse Time 2.98s (± 0.72%) 3.00s (± 0.65%) ~ 2.98s 3.03s p=0.141 n=6
Bind Time 1.49s (± 0.93%) 1.47s (± 7.50%) ~ 1.25s 1.53s p=0.121 n=6
Check Time 9.27s (± 0.42%) 9.35s (± 1.19%) ~ 9.27s 9.57s p=0.092 n=6
Emit Time 5.32s (± 0.55%) 5.34s (± 0.47%) ~ 5.30s 5.37s p=0.225 n=6
Total Time 19.04s (± 0.37%) 19.16s (± 0.22%) +0.12s (+ 0.62%) 19.11s 19.21s p=0.013 n=6
material-ui - node (v18.15.0, x64)
Memory used 510,182k (± 0.01%) 517,152k (± 0.00%) +6,971k (+ 1.37%) 517,122k 517,170k p=0.005 n=6
Parse Time 3.93s (± 0.30%) 3.97s (± 0.38%) +0.04s (+ 1.06%) 3.96s 4.00s p=0.005 n=6
Bind Time 1.46s (± 1.04%) 1.51s (± 0.72%) +0.05s (+ 3.78%) 1.50s 1.53s p=0.005 n=6
Check Time 25.37s (± 0.32%) 25.42s (± 0.44%) ~ 25.26s 25.57s p=0.468 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 30.76s (± 0.27%) 30.90s (± 0.37%) ~ 30.73s 31.06s p=0.065 n=6
mui-docs - node (v18.15.0, x64)
Memory used 1,744,924k (± 0.00%) 1,750,658k (± 0.00%) +5,734k (+ 0.33%) 1,750,580k 1,750,730k p=0.005 n=6
Parse Time 9.63s (± 0.64%) 9.83s (± 0.46%) +0.20s (+ 2.04%) 9.76s 9.88s p=0.005 n=6
Bind Time 3.46s (± 0.46%) 2.81s (± 0.85%) 🟩-0.65s (-18.87%) 2.79s 2.85s p=0.005 n=6
Check Time 82.05s (± 0.38%) 84.60s (± 2.09%) +2.55s (+ 3.11%) 82.49s 87.27s p=0.008 n=6
Emit Time 0.19s (± 0.00%) 0.19s (± 4.29%) ~ 0.19s 0.21s p=0.176 n=6
Total Time 95.34s (± 0.34%) 97.44s (± 1.82%) +2.10s (+ 2.20%) 95.36s 100.16s p=0.031 n=6
self-build-src - node (v18.15.0, x64)
Memory used 2,401,405k (± 0.02%) 2,491,144k (± 0.01%) +89,739k (+ 3.74%) 2,490,674k 2,491,672k p=0.005 n=6
Parse Time 5.00s (± 1.03%) 5.26s (± 0.97%) 🔻+0.26s (+ 5.30%) 5.20s 5.35s p=0.005 n=6
Bind Time 1.90s (± 1.13%) 1.71s (± 0.86%) 🟩-0.19s (-10.07%) 1.69s 1.73s p=0.005 n=6
Check Time 33.91s (± 0.13%) 34.41s (± 0.29%) +0.49s (+ 1.45%) 34.24s 34.52s p=0.005 n=6
Emit Time 2.67s (± 0.73%) 2.61s (± 2.41%) ~ 2.51s 2.68s p=0.065 n=6
Total Time 43.50s (± 0.17%) 44.01s (± 0.30%) +0.51s (+ 1.18%) 43.83s 44.15s p=0.005 n=6
self-compiler - node (v18.15.0, x64)
Memory used 416,563k (± 0.00%) 422,214k (± 0.01%) +5,651k (+ 1.36%) 422,167k 422,256k p=0.005 n=6
Parse Time 4.11s (± 0.29%) 4.04s (± 1.07%) -0.07s (- 1.66%) 3.98s 4.09s p=0.005 n=6
Bind Time 1.58s (± 0.74%) 1.73s (± 1.01%) 🔻+0.15s (+ 9.82%) 1.71s 1.76s p=0.005 n=6
Check Time 22.37s (± 0.53%) 22.39s (± 0.56%) ~ 22.24s 22.54s p=0.936 n=6
Emit Time 1.66s (± 1.55%) 1.66s (± 0.59%) ~ 1.64s 1.67s p=0.438 n=6
Total Time 29.72s (± 0.41%) 29.84s (± 0.49%) ~ 29.66s 29.98s p=0.253 n=6
vscode - node (v18.15.0, x64)
Memory used 2,907,995k (± 0.00%) 2,909,876k (± 0.01%) +1,881k (+ 0.06%) 2,909,542k 2,910,059k p=0.005 n=6
Parse Time 15.93s (± 0.48%) 16.37s (± 0.29%) +0.44s (+ 2.75%) 16.31s 16.42s p=0.005 n=6
Bind Time 5.08s (± 0.72%) 5.07s (± 0.29%) ~ 5.05s 5.09s p=0.568 n=6
Check Time 88.20s (± 0.48%) 88.17s (± 0.34%) ~ 87.81s 88.49s p=0.936 n=6
Emit Time 24.46s (± 8.12%) 25.46s (± 9.53%) 🔻+1.00s (+ 4.07%) 23.75s 28.66s p=0.037 n=6
Total Time 133.68s (± 1.67%) 135.08s (± 1.95%) +1.40s (+ 1.04%) 133.09s 138.61s p=0.045 n=6
webpack - node (v18.15.0, x64)
Memory used 409,053k (± 0.01%) 415,675k (± 0.02%) +6,623k (+ 1.62%) 415,553k 415,795k p=0.005 n=6
Parse Time 4.81s (± 0.40%) 4.89s (± 0.92%) +0.08s (+ 1.70%) 4.83s 4.94s p=0.008 n=6
Bind Time 2.07s (± 0.83%) 2.05s (± 0.37%) -0.02s (- 0.97%) 2.04s 2.06s p=0.024 n=6
Check Time 20.91s (± 0.48%) 21.09s (± 0.27%) +0.18s (+ 0.84%) 21.02s 21.18s p=0.010 n=6
Emit Time 0.00s (± 0.00%) 0.00s (± 0.00%) ~ 0.00s 0.00s p=1.000 n=6
Total Time 27.80s (± 0.40%) 28.04s (± 0.27%) +0.24s (+ 0.86%) 27.96s 28.13s p=0.005 n=6
xstate - node (v18.15.0, x64)
Memory used 513,523k (± 0.02%) 531,559k (± 0.02%) +18,035k (+ 3.51%) 531,411k 531,696k p=0.005 n=6
Parse Time 3.95s (± 0.59%) 3.99s (± 0.75%) ~ 3.96s 4.04s p=0.063 n=6
Bind Time 1.88s (± 1.10%) 1.67s (± 0.73%) 🟩-0.21s (-11.20%) 1.65s 1.68s p=0.004 n=6
Check Time 3.43s (± 0.47%) 3.62s (± 0.30%) 🔻+0.19s (+ 5.59%) 3.60s 3.63s p=0.004 n=6
Emit Time 0.09s (± 6.44%) 0.08s (± 4.99%) ~ 0.08s 0.09s p=0.213 n=6
Total Time 9.35s (± 0.31%) 9.36s (± 0.17%) ~ 9.34s 9.39s p=0.618 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • Angular - node (v18.15.0, x64)
  • Compiler-Unions - node (v18.15.0, x64)
  • Monaco - node (v18.15.0, x64)
  • TFS - node (v18.15.0, x64)
  • material-ui - node (v18.15.0, x64)
  • mui-docs - node (v18.15.0, x64)
  • self-build-src - node (v18.15.0, x64)
  • self-compiler - node (v18.15.0, x64)
  • vscode - node (v18.15.0, x64)
  • webpack - node (v18.15.0, x64)
  • xstate - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

tsserver

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
Compiler-UnionsTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,366ms (± 0.60%) 2,353ms (± 0.60%) ~ 2,337ms 2,371ms p=0.230 n=6
Req 2 - geterr 5,625ms (± 0.72%) 5,149ms (± 0.52%) 🟩-477ms (- 8.48%) 5,124ms 5,195ms p=0.005 n=6
Req 3 - references 319ms (± 1.28%) 295ms (± 1.93%) 🟩-24ms (- 7.53%) 287ms 299ms p=0.005 n=6
Req 4 - navto 272ms (± 0.00%) 228ms (± 0.36%) 🟩-44ms (-16.05%) 228ms 230ms p=0.002 n=6
Req 5 - completionInfo count 1,357 (± 0.00%) 1,357 (± 0.00%) ~ 1,357 1,357 p=1.000 n=6
Req 5 - completionInfo 87ms (± 0.86%) 77ms (± 0.00%) 🟩-10ms (-11.66%) 77ms 77ms p=0.003 n=6
CompilerTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,501ms (± 1.09%) 2,502ms (± 1.10%) ~ 2,464ms 2,529ms p=1.000 n=6
Req 2 - geterr 4,183ms (± 1.66%) 3,872ms (± 0.33%) 🟩-311ms (- 7.43%) 3,858ms 3,891ms p=0.005 n=6
Req 3 - references 332ms (± 0.30%) 297ms (± 0.72%) 🟩-35ms (-10.45%) 294ms 300ms p=0.005 n=6
Req 4 - navto 297ms (± 0.69%) 240ms (± 2.23%) 🟩-57ms (-19.29%) 229ms 243ms p=0.003 n=6
Req 5 - completionInfo count 1,519 (± 0.00%) 1,519 (± 0.00%) ~ 1,519 1,519 p=1.000 n=6
Req 5 - completionInfo 73ms (± 1.86%) 70ms (± 6.07%) ~ 68ms 79ms p=0.059 n=6
xstateTSServer - node (v18.15.0, x64)
Req 1 - updateOpen 2,637ms (± 0.34%) 2,594ms (± 0.65%) -43ms (- 1.62%) 2,571ms 2,615ms p=0.005 n=6
Req 2 - geterr 1,736ms (± 1.93%) 1,559ms (± 4.07%) 🟩-177ms (-10.22%) 1,522ms 1,687ms p=0.005 n=6
Req 3 - references 115ms (±10.74%) 97ms (± 8.96%) 🟩-17ms (-14.99%) 92ms 115ms p=0.020 n=6
Req 4 - navto 367ms (± 0.87%) 357ms (± 1.67%) -11ms (- 2.86%) 350ms 367ms p=0.016 n=6
Req 5 - completionInfo count 2,079 (± 0.00%) 2,079 (± 0.00%) ~ 2,079 2,079 p=1.000 n=6
Req 5 - completionInfo 310ms (± 1.45%) 290ms (± 0.92%) 🟩-20ms (- 6.30%) 287ms 295ms p=0.005 n=6
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • CompilerTSServer - node (v18.15.0, x64)
  • Compiler-UnionsTSServer - node (v18.15.0, x64)
  • xstateTSServer - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

startup

Comparison Report - baseline..pr
Metric baseline pr Delta Best Worst p-value
tsc-startup - node (v18.15.0, x64)
Execution time 154.51ms (± 0.17%) 229.76ms (± 0.17%) 🔻+75.25ms (+48.70%) 228.44ms 234.80ms p=0.000 n=600
tsserver-startup - node (v18.15.0, x64)
Execution time 227.90ms (± 0.14%) 228.66ms (± 0.15%) +0.76ms (+ 0.33%) 227.18ms 231.48ms p=0.000 n=600
tsserverlibrary-startup - node (v18.15.0, x64)
Execution time 223.67ms (± 0.14%) 224.41ms (± 0.16%) +0.75ms (+ 0.33%) 222.83ms 231.38ms p=0.000 n=600
typescript-startup - node (v18.15.0, x64)
Execution time 222.22ms (± 0.16%) 223.03ms (± 0.18%) +0.81ms (+ 0.37%) 221.50ms 230.11ms p=0.000 n=600
System info unknown
Hosts
  • node (v18.15.0, x64)
Scenarios
  • tsc-startup - node (v18.15.0, x64)
  • tsserver-startup - node (v18.15.0, x64)
  • tsserverlibrary-startup - node (v18.15.0, x64)
  • typescript-startup - node (v18.15.0, x64)
Benchmark Name Iterations
Current pr 6
Baseline baseline 6

Developer Information:

Download Benchmarks

@jakebailey
Copy link
Member Author

Pretty good, not too far off tsc anymore.

@jakebailey
Copy link
Member Author

Closing, since we'll be able to test this in a different way now without that patch.

@jakebailey jakebailey closed this Apr 4, 2024
@jakebailey jakebailey deleted the mono-services-plus-tsc branch April 4, 2024 22:58
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers
No reviews
Assignees

@jakebailey jakebailey

Labels
Author: Team For Uncommitted Bug PR for untriaged, rejected, closed or missing bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@jakebailey @typescript-bot