Implement "Bit Watch" with support of TS Project References

[davidfirst]

Description

Currently, bit watch is slow. It runs the whole "compile" process for every change.
For TS, it needs to run the compile on the dependencies/dependents as well.

Running tsc -w on each one of the capsules is problematic because we'll end up with lots of processes and high memory usage.

Describe the solution you'd like

When the compiler is typescript, take advantage of the Project References to speed up the process.
Once implemented, one process of tsc --build -w will be running and it'll watch all capsules changes.

Additional context

Step 1: the user is running bit watch

• watch extension loads all components in the workspace.
• calls to a new Compile extension API getWatchProccesses(components: Component[]).
• the compile extension aggregates all components that their specific compiler implements a new API call watchMultiple(capsulePaths: string[]). It aggregates them by the compiler version.
• typescript compilers that support Project References, will implement that method.
• for each one of the aggregated group, it runs watchMultiple.
• the watchMultiple, creates a tsconfig.json file with the project references in the tmp dir. The file name should be a hash of the capsules-paths with tsconifg.json suffix. (so one group won't collide with another and only one file is created per group).
  e.g.: a file abcdef.tsconfig.json can have the following:

{
  "files": [],
  "include": [],
  "references": [
    { "path": "<capsule-root>/bit.core_create@0.0.5" },
    { "path": "<capsule-root>bit.core_pack@0.0.5" }
  ]
}

• At this phase, it's possible to run tsc --build abcdef.tsconfig.json -w from the CLI, and it watches all the directories mentioned in the references array. (each one of the directories above should have its own tsconfig.json file).
• for each aggregated group, Compile extension creates a new childProcess that runs the tsc command above and return them all to the Watch extension.
• for non-ts components or legacy components, it doesn't do anything.

Step 2: user changes a component file

• bit watch knows what component-id is it, it reloads the consumer+bitmap and the changed component. The reason for reloading them all is that a component might be tagged in the meanwhile, and the component in-memory is the old version that associated with the older capsule.

Step 3A: user changes a Non-TS component file

• bit watch runs the "compile" extension with the component-id.

Step 3B: user changes a TS component file

• Reloading the component probably takes care of re-writing the files into the capsule. If not, manually write the changed files into the capsule.
• If the capsule paths were changed, kill the corresponding watch process and re-create it.
• Once the files are written into the capsule, the "tsc -w" is triggered and it writes the dists into the capsules.

Performance

Tested on the harmony/dogfooding branch with our extensions. (171 files, 5,562 LOC, 25 extensions).
The watch is still not as fast as I would expect. It takes around 2.5 seconds on v3.8 and about 1.5 seconds when using the new 3.9 version.

When the tsc -w is running on the workspace (bit-bin), the watch takes less than a second to complete. Interestingly enough, when using tsc --build -w, it takes 3 seconds.
I don't understand why --build is slower, it should be faster as it uses an incremental build.
It has been asked here microsoft/TypeScript#25600 (comment) and I hope it'll be answered.
Understanding this will help us to improve the performance running tsc --build -w on the capsules.

[davidfirst]

After a POC, this is not going to work well.
I tried the structure suggested in #2582 (comment), which symlinks the implementation files to the capsule "src" dir. It helped the performance dramatically. However, I found other issues around the development experience.
(I'm working on VSCode, so I'll give specific examples related to this IDE, but it's relevant to others as well).

1. In order to debug the component in the VSCode, you have to add the capsule dist directory as the "outFiles".

 "outFiles": [  "/Users/davidfirst/Library/Caches/Bit/capsules/f8628c4b69f6257e79f1bf3cb211a966a0022240/comp1/dist/**/*.js",
      ]

This dist on the capsule has the SourceMap, which looks something like this:

{"version":3,"file":"index.js","sourceRoot":"/path/to/the/workspace/comp1/","sources":["index.ts"],"names":[],"mappings":";;;;;AAAA,uDAA+B;AAC/B,OAAO,CAAC,GAAG,CAAC,SAAS,CAAC,CAAA;AACtB,kBAAe,GAAG,EAAE,CAAC,YAAY,GAAG,eAAK,EAAE,CAAC;AAC5C"}%

As you can see, the source-map has the sourceRoot prop pointing to the workspace, this way, the debugger knows to listen to the breakpoint on the source code on the workspace.
The thing is that the developer must be aware of the capsule dir and add it to the launch.json of VSCode.

2. It must be done per component. So lots of work for developers (Pointing the outFiles to the capsules root doesn't work. The workspace doesn't have the same structure as the capsules and the source-map needs to have the sourceRoot of its own component directory.)

3. components that don't have any compiler set (plain javascript) will need dist directory with a copy of the source files. Otherwise, it's not possible to run them from the capsule because the src is symlinked to the workspace and the workspace doesn't have the node-modules. The problem is that this dist obviously doesn't have any source-map, so the debugger is unable to connect it to the source on the workspace.
   This issue specifically can be already resolved if the solution to the workspace-resolution will be installing the node-modules in the components dir (which is the route we're going to take).

I discussed this with @ranm8 , and the conclusion is, in short:
For development, the watch will be done on the workspace for the best debugging and compiling experience. For release/publish/tag the compile will be executed on the capsule for the best isolation and to be consistent between author and imported.
To get it to work on the workspace, a package.json will be saved in the node_modules directory of the component and all implementation files will be symlinked to the node_modules. The dists will be saved inside the component dir and will be symlinked to the node_modules.
We checked it quickly and the debugging is working great on both, the browser and in VSCode.
@GiladShoham , FYI.

[davidfirst]

Another POC I tried, similar to the suggestion in the comment above with an improvement - the dists sit in the node_modules dir and no need to link them to the workspace component's dir.
I tried it with the programmatic API and project references, it seems to be working well.
The sourceMap are generated correctly so the debugging on VSCode is working. The navigation in the IDE is working as well due to the addition of "declarationMap": true,.
The only issue is when there is a compilation error, it shows the file in node-modules and not in the workspace, but that's not so bad because 1) clicking on the file in the terminal point to the right place. 2) we can parse the error and change the file path.
Here is the final tsconfig.json that is written to the node_modules of the component:

{
  "compilerOptions": {
    "outDir": "dist",
    "sourceMap": true,
    "sourceRoot": "/path/to/the/root/component/dir",
    "declarationMap": true,
    "preserveSymlinks": true,
    "composite": true
  },
  "exclude": [
    "dist"
  ]
}

And this is the gist of the compilation process using the ts API:

function compile(componentDirs: string[], options: ts.CompilerOptions): void {
  const solutionBuilderHost = ts.createSolutionBuilderHost();
  const solutionBuilder = ts.createSolutionBuilder(solutionBuilderHost, componentDirs, { verbose: true });
  solutionBuilder.clean();
  const result = solutionBuilder.build();
  console.log("result", result)
}

Similar API exists for watch. Just replace createSolutionBuilder with createSolutionBuilderWithWatch.

[davidfirst]

After discussing with Ran, we can improve it.
For the development environment, no need for the d.ts files, so no need to use the project references. We can use the ts.transpileModule API to transpile files one by one. No need to be aware of the dependents/dependencies because no need for the d.ts files.

function compileFile() {
  const source = `import comp2 from '@bit/comp2';
  export default () => 'comp1 and ' + comp2();`;

  const result = ts.transpileModule(source, {
    compilerOptions: {
      module: ts.ModuleKind.CommonJS,
      sourceMap: true,
      sourceRoot: '/path/to/the/component/root'
    },
    fileName: 'index.ts',
    reportDiagnostics: true
  });

  console.log(JSON.stringify(result));
}

It's wip on #2698.

[davidfirst]

In the branch above (harmony/compile-multiple) I was trying a POC of using the project references for the compilation on the capsules.
In that branch, the compilation is done successfully but the dists are not saved.
The reason I stopped with the implementation, for now, is because I didn't see a big performance improvement.
For 100 components (1 file, 2 LOC, 1 dep each), the compilation on the capsules takes 2 minutes. With the improvements here, it takes 1:40. It's not bad, but not sure it worths the inconsistency we might get when running the compilation through Flows.

[davidfirst]

Completed.
We ended up with an implementation similar to the last comment.
During development, the compiler and watch are done on the workspace.
For the release, the compilation is done on capsules and then the Project References is used to speed up the process.