Explore "transpile only" build

[jrieken]

For CI builds we should explore the use of a fast transpiler, like SWC or esbuild. This would enable use to run (blocking) transpile, then in parallel run tests and TS type checking.

SWC transpile command: npx swc --config-file .swcrc src --copy-files --out-dir out

---

The joh/swc-branch has some changes that explore to use SWC for this. TODOs

• support for AMD - SWC just does that, great!
• we heavily rely on TS decorators for our injection story and need that to be supported. Good news is that latest SWC 1.2.186 has this fixed
• SWC doesn't hoist const enums swc/issues/3648 which isn't ideal for perf but also exposes some cyclic module dependencies that we have. We need to fix
  • Cyclic dependency: vs/workbench/contrib/markers/browser/constants and vs/workbench/contrib/markers/browser/markersTreeViewer #150291
  • Cyclic dependency: vs/editor/common/languages and vs/editor/common/tokenizationRegistry #150176
  • Cyclic dependency: vs/editor/common/viewLayout/lineDecorations and vs/editor/common/viewLayout/viewLineRenderer #150177
  • Cyclic dependency: vs/workbench/contrib/tasks/common/tasks and vs/workbench/contrib/tasks/common/taskService #150178
• The --copy-files flag doesn't seem to copy JS files (unclear if that's because they are compile-ignored or always seen as compilable resource)
• Workaround export import bug swc/issues/4481
• Local require-function is missing from AMD modules, swc/issues/4718
  • Local AMD require function not always called require swc-project/swc#4797
  • Local AMD require call sometimes missing swc-project/swc#4798
• Using a namespace before its declaration fails, swc/issues/4897
• Issue with import-require mixing, either avoid or wait for import foo = require should be normal AMD dependency swc-project/swc#4898
• Static factory function in decorated class uses un-decorated ctor swc-project/swc#4899

[kwonoj]

👋 Just want to say excited to see vscode explores swc as transpiler and happy to try to support as much. Please drop by swc repo if there's something to followup (Yes, saw one issue filed already)

[kwonoj]

swc-project/swc#4481 is updated.

[jrieken]

@kwonoj Awesome. What's your release cycle like? We are now heading into endgame and I'll be able to pick this up early June again. Are those fixes published by then?

[kwonoj]

We should able to. @kdy1 , if we didn't publish new one can we do it for @swc/core?

[kdy1]

I triggered a publish of a new version

[jrieken]

@kdy1 @kwonoj Thanks so far. With 1.2.192 things are looking better but I ran into two further bugs with AMD's local require-function: swc-project/swc#4798 and swc-project/swc#4797. Let me know if you need further clarification or help. Thanks a ton!

[kwonoj]

Thanks, I'll try repro.

[kwonoj]

swc-project/swc#4800 is updated for 4798/4797.

[jrieken]

Thanks - I am OOO for a bit and I'll pick this up again early June

[jrieken]

@kdy1 @kwonoj Any chance for releasing a new version of SWC with your latest fixes? We should be pretty close.. With 1.2.197 97% of our unit tests already pass

[kdy1]

Currently CI of swc is broken due to recent changes of github actions, but it's almost fixed by another team member.
Once he sends a PR for fixing CI scripts, I'll publish a new verion.

[kdy1]

The new version is published!

https://github.com/swc-project/swc/actions/runs/2479477756

[kwonoj]

👋 Casually checking in (not urging or pushing anything), if there's some progress or if there are any blockers in upstream swc land we can try to fix forward?

[jrieken]

@kwonoj Great timing! Two days back I have merged #159300. For CI/CD we now use SWC to transpile our "client" bits (basically all of VS Code but not yet its builtin extensions). Look out for transpile-client-swc in our PR checks (random sample).

Random observations and things that can be improved

• I ended up with two separate config files (one for amd and one with no module system) because I couldn't figure out how to achieve this in one config file 🙈. We had discussed this but I couldn't get it working with a single file. Maybe you have more insights
• Transpile is part of our ancient gulp pipeline but we don't use SWC as library- we simply shell out to swc-cli. That's good because we don't need to pipe files around but ideally we shell out to the native executable directly so that we don't use JS to write files. @kwonoj @kdy1 Do you have any clue what the perf difference would be (write files from rust instead of JS)?
• Along the way we also added a transpiler that uses TypeScript's transpileModule-API. We have this as fallback/alternative just in case something goes wrong (new TS version that SWC hasn't adopted yet, etc). SWC is faster but "only" twice as fast, do you think more could be gained with SWC? Like avoid swc-cli or swc-core and use the binaries directly?

@kdy1 @kwonoj With all that said: a very big "Thank You" for SWC and for the instant help and fixes. We are all very excited about this and hope to get involved more deeply (use SWC to watch-transpile on our dev machines etc)

[kwonoj]

I couldn't figure out how to achieve this in one config file

Let's retry this. If you don't mind share some minimal repro you'd like to try (and not working), I'll try to peek.

ideally we shell out to the native executable directly so that we don't use JS to write files

Correct, native cli should be more performant. Not cause of writing performance, there are few things we may consider

1. You don't need to cost spin node.js process: npx runs node.js script under npm's env, and then swc-cli itself runs with node.js again. Native cli won't need those bootstrap performance.

2. Native cli can parallelize multiple inputs when performing transform, without back and forth v8's async event loop

Currently, most of @swc/core native bindings ships with experimental native cli. You may give it a go, and if there's some missing features / bugs we can try to fix it forward.

SWC is faster but "only" twice as fast, do you think more could be gained with SWC

This sounds odd. When I tried transform performance with some large scale codes, even though it was compared with babel it was much visibly faster. I guess we could try to create some isolated cases to dig deeper, probably using either native cli / or node.js bindings api directly. It is totally possible js based cli's overhead cuts off some perf gains.

[kdy1]

There are some code which make handling of small files faster but have exponential time complexity.

What's your target/.swcrc and input?

[jrieken]

Let's retry this. If you don't mind share some minimal repro you'd like to try (and not working), I'll try to peek.

Repos with a .swcrc-file that is an array, not object. I can file a real issue, the task I want to solve is that for one source file we need a different module system - all files should be compiled with AMD, except for vs/base/worker/workerMain.ts

EDIT: created swc-project/swc#5720 to follow up on this

[jrieken]

This sounds odd. When I tried transform performance with some large scale codes, even though it was compared with babel it was much visibly faster. I guess we could try to create some isolated cases to dig deeper, probably using either native cli / or node.js bindings api directly. It is totally possible js based cli's overhead cuts off some perf gains.

Yeah, that's what I though. In all fairness, we have custom code around transpileModule which uses NodeJS worker threads and other perf tricks. I'll explore how to use the native cli directly. Does @swc/core come with an API that exposes the path of the actually binary? (so that this is easy across platforms)

[kwonoj]

Does @swc/core come with an API that exposes the path of the actually binary? (so that this is easy across platforms)

No, but calculation is rather straightforward: https://github.com/swc-project/swc/blob/7a6a43576a3388cb07d6c8923b9118228b8e09f0/node-swc/src/run_swcx.ts#L39-L80

You can try to copy-paste with small modifications, maybe. Or you could do npx swcx, but I do not recommend since it'll require spinning npm for the npx execution then it goes to another spawning to swc cli.

[jrieken]

You can try to copy-paste with small modifications, maybe

This looks promising: I am running /Users/jrieken/Code/vscode/node_modules/@swc/core-darwin-arm64/swc compile --config-file build/lib/swc/.swcrc-amd --out-dir out3 --source-root src/ src/**/*.ts. I do get output but it is rooted at out3/src/vs instead of out3/vs. Do I need to pass another value to source-root or how is that meant to work?

[kwonoj]

Do I need to pass another value to source-root or how is that meant to work?

I believe it's simple lacks of features (after all, native cli is still not yet feature-complete). If you don't mind file an issue in upstream swc, I'll try to assign myself.

[jrieken]

With #160274 we now have SwcTranspiler which we use to transpile the client and all extension bits. This transpiler can be used interchangeably with the TscTranspiler but is ~3 times faster.