feat: faster, lazy-friendly runtime loading (#4181)

Instead of eagerly loading each type from the assemblies in order to annotate the constructors with jsii fully qualified type names (FQNs), leverage the `jsii.rtti` data that is injected by the `jsii` compiler since release `1.19.0`.

This should make the jsii runtimes friendlier to large libraries that include lazy-loading provisions, such as the `aws-cdk-lib`.

---

In addition to this, `JSII_RUNTIME_PACKAGE_CACHE` was flipped from opt-in to opt-out (set it to any value other than `enabled` to disable), and the `@jsii/runtime` entry point now sets `--preserve-symlinks` so that we can symbolically link packages from the cache instead of copying them around, which is significantly faster.

---

Finally, the jsii kernel had not opted out of the assembly validation feature, which is redundant in the majority of scenarios, and is quite time-consuming (~500ms for `aws-cdk-lib`)... So also opting out, but allowing users to opt-back-in via an environment variable.

---

Example on a "simple" repro via `aws-cdk-lib`:

Before:
```
[@jsii/kernel:timing] tar.extract(<redacted>/.venv-vanilla/lib/python3.11/site-packages/aws_cdk/_jsii/aws-cdk-lib@2.87.0.jsii.tgz) => <redacted>/jsii-kernel-Xxp43L/node_modules/aws-cdk-lib: 1.909s
[@jsii/kernel:timing] loadAssemblyFromPath(<redacted>/jsii-kernel-Xxp43L/node_modules/aws-cdk-lib): 383.8ms
[@jsii/kernel:timing] require(<redacted>/jsii-kernel-Xxp43L/node_modules/aws-cdk-lib): 630.081ms
[@jsii/kernel:timing] registerAssembly({ name: aws-cdk-lib, types: 10957 }): 8.452ms
[@jsii/kernel:timing] load({
  "name": "aws-cdk-lib",
  "version": "2.87.0",
  "tarball": "<redacted>/.venv-vanilla/lib/python3.11/site-packages/aws_cdk/_jsii/aws-cdk-lib@2.87.0.jsii.tgz",
  "api": "load"
}): 2.933s
```

After:
```
[@jsii/kernel:timing] tar.extract(<redacted>/.venv-lazy/lib/python3.11/site-packages/aws_cdk/_jsii/aws-cdk-lib@2.87.0.jsii.tgz) => <redacted>/jsii-kernel-eAOMah/node_modules/aws-cdk-lib: 12.247ms
[@jsii/kernel:timing] loadAssemblyFromPath(<redacted>/jsii-kernel-eAOMah/node_modules/aws-cdk-lib): 388.388ms
[@jsii/kernel:timing] require(<redacted>/jsii-kernel-eAOMah/node_modules/aws-cdk-lib/lazy-index.js): 132.801ms
[@jsii/kernel:timing] registerAssembly({ name: aws-cdk-lib, types: 10957 }): 0.009ms
[@jsii/kernel:timing] load({
  "name": "aws-cdk-lib",
  "version": "2.87.0",
  "tarball": "<redacted>/.venv-lazy/lib/python3.11/site-packages/aws_cdk/_jsii/aws-cdk-lib@2.87.0.jsii.tgz",
  "api": "load"
}): 537.449ms
```

---

By submitting this pull request, I confirm that my contribution is made under the terms of the [Apache 2.0 license].

[Apache 2.0 license]: https://www.apache.org/licenses/LICENSE-2.0

CONTRIBUTING.md

@@ -89,6 +89,71 @@ Each one of these scripts can be executed either from the root of the repo using
 `npx lerna run <script> --scope <package>` or from individual modules using
 `yarn <script>`.
 
+#### Reproducting Bugs (Test-Driven Solving)
+
+Troubleshooting bugs usually starts with adding a new test that demonstrates the
+faulty behavior, then modifying implementations until the test passes.
+
+The `jsii-calc` and `@scope/*` packages are used to test expected brhavior from
+the compiler (note that the [aws/jsii-compiler](github.com/aws/jsii-compiler)
+repository as a separate copy of these under the `fixtures` directory), as well
+as downstream tooling (`jsii-pacmak`, `jsii-rosetta`, etc...). Each language
+runtime has its own test suite, within which is a _compliance_ suite that tests
+the same behaviors in all languages, and which should contain tests related to
+behavior that isn't strictly specific to the given language.
+
+The `yarn test:update` script in each package runs all tests and updates
+snapshots when necessary. It is usually necessary to run this script at least in
+`jsii-pacmak` and `jsii-reflect` after changing code in the `jsii-calc` or
+`@scope/*` packages.
+
+#### Debugging runtime behavior
+
+Cross-language runtime behavior can be challenging to debug, as data is passed
+across process boundaries through Inter-Process Communication (IPC) channels.
+Further complicating things, the `@jsii/runtime` library packaged in the various
+language runtimes is bundled (by `webpack`), which can make the Javascript
+runtime code more complicated to follow.
+
+Setting various environment variables can help understanding what is happening
+better:
+
+- `JSII_DEBUG=1` turns on verbose debug logging, which will cause the program to
+  emit extensive IPC tracing information to `STDERR`. This information can help
+  identify where things start to behave in unexpected ways, but can be a little
+  difficult to digest... One may want to refer to the [kernel API][kernel-api]
+  documentation to make sense of those traces.
+
+- `JSII_DEBUG_TIMING=1` turns on specific timing information from the
+  `@jsii/kernel` high level API processing, which can be useful to narrow down
+  the possible causes for performance issues.
+
+- `JSII_RUNTIME` can be set to point to the `bin/jsii-runtime` script within the
+  `@jsii/runtime` package in order to use a local, non-`webpack`ed version of
+  the runtime program. This can be particularly helpful when trying to diagnose
+  a problem within a debugger session.
+
+- `NODE_OPTIONS` can be used to configure specific behaviors of the underlying
+  `node` runtime, such as specifying `--inspect-brk` to cause the node process
+  to wait for a debugger to attach before proceeding. This is useful to attach
+  Node dev tools to the runtime as it starts in order to use its debugger.
+
+The [Visual Studio Code](https://code.visualstudio.com) _JavaScript Debug
+Terminal_ feature can be particularly useful paired with appropriate
+`JSII_RUNTIME` setting to run arbitrary jsii programs, automatically attaching
+the VSCode debugger at startup. These terminals inject a specially crafted
+`NODE_OPTIONS` variable that allows the VSCode debugger to consistently attach
+to all `node` processes spawned within its context, including child processes
+(which can be problematic when running with `--inspect-brk`, as the default
+debugger interface's port can only be used by one process at a time).
+
+Finally, the `debugger` Javascript statement can be added anywhere in the
+runtime code or tested libraries in order to cause debuggers (if attached) to
+pause. This can be easier (and more reliable) to set up than traditional
+conditional break points.
+
+[kernel-api]: https://aws.github.io/jsii/specification/3-kernel-api/
+
 #### Linting & Formatting
 
 Eslint and Prettier are used to lint and format our typescript code. The `lint`

packages/@jsii/kernel/src/kernel.ts

@@ -43,6 +43,10 @@ export class Kernel {
    * Set to true for timing data to be emitted.
    */
   public debugTimingEnabled = false;
+  /**
+   * Set to true to validate assemblies upon loading (slow).
+   */
+  public validateAssemblies = false;
 
   readonly #assemblies = new Map<string, Assembly>();
   readonly #objects = new ObjectTable(this.#typeInfoForFqn.bind(this));
@@ -69,6 +73,7 @@ export class Kernel {
     this.#serializerHost = {
       objects: this.#objects,
       debug: this.#debug.bind(this),
+      isVisibleType: this.#isVisibleType.bind(this),
       findSymbol: this.#findSymbol.bind(this),
       lookupType: this.#typeInfoForFqn.bind(this),
     };
@@ -147,10 +152,10 @@ export class Kernel {
     }
 
     // read .jsii metadata from the root of the package
-    let assmSpec;
+    let assmSpec: spec.Assembly;
     try {
       assmSpec = this.#debugTime(
-        () => spec.loadAssemblyFromPath(packageDir),
+        () => spec.loadAssemblyFromPath(packageDir, this.validateAssemblies),
         `loadAssemblyFromPath(${packageDir})`,
       );
     } catch (e: any) {
@@ -159,10 +164,15 @@ export class Kernel {
       );
     }
 
+    // We do a `require.resolve` call, as otherwise, requiring with a directory will cause any `exports` from
+    // `package.json` to be ignored, preventing injection of a "lazy index" entry point.
+    const entryPoint = this.#require!.resolve(assmSpec.name, {
+      paths: [this.#installDir!],
+    });
     // load the module and capture its closure
     const closure = this.#debugTime(
-      () => this.#require!(packageDir),
-      `require(${packageDir})`,
+      () => this.#require!(entryPoint),
+      `require(${entryPoint})`,
     );
     const assm = new Assembly(assmSpec, closure);
     this.#debugTime(
@@ -333,7 +343,7 @@ export class Kernel {
       throw new JsiiFault(`${method} is an async method, use "begin" instead`);
     }
 
-    const fqn = jsiiTypeFqn(obj);
+    const fqn = jsiiTypeFqn(obj, this.#isVisibleType.bind(this));
     const ret = this.#ensureSync(
       `method '${objref[TOKEN_REF]}.${method}'`,
       () => {
@@ -420,7 +430,7 @@ export class Kernel {
       throw new JsiiFault(`Method ${method} is expected to be an async method`);
     }
 
-    const fqn = jsiiTypeFqn(obj);
+    const fqn = jsiiTypeFqn(obj, this.#isVisibleType.bind(this));
 
     const promise = fn.apply(
       obj,
@@ -568,6 +578,21 @@ export class Kernel {
   #addAssembly(assm: Assembly) {
     this.#assemblies.set(assm.metadata.name, assm);
 
+    // We can use jsii runtime type information from jsii 1.19.0 onwards... Note that a version of
+    // 0.0.0 means we are assessing against a development tree, which is newer...
+    const jsiiVersion = assm.metadata.jsiiVersion.split(' ', 1)[0];
+    const [jsiiMajor, jsiiMinor, _jsiiPatch, ..._rest] = jsiiVersion
+      .split('.')
+      .map((str) => parseInt(str, 10));
+    if (
+      jsiiVersion === '0.0.0' ||
+      jsiiMajor > 1 ||
+      (jsiiMajor === 1 && jsiiMinor >= 19)
+    ) {
+      this.#debug('Using compiler-woven runtime type information!');
+      return;
+    }
+
     // add the __jsii__.fqn property on every constructor. this allows
     // traversing between the javascript and jsii worlds given any object.
     for (const fqn of Object.keys(assm.metadata.types ?? {})) {
@@ -869,7 +894,7 @@ export class Kernel {
     methodInfo: spec.Method,
   ) {
     const methodName = override.method;
-    const fqn = jsiiTypeFqn(obj);
+    const fqn = jsiiTypeFqn(obj, this.#isVisibleType.bind(this));
     const methodContext = `${methodInfo.async ? 'async ' : ''}method${
       fqn ? `${fqn}#` : methodName
     }`;
@@ -1029,7 +1054,7 @@ export class Kernel {
     return curr;
   }
 
-  #typeInfoForFqn(fqn: string): spec.Type {
+  #typeInfoForFqn(fqn: spec.FQN): spec.Type {
     const components = fqn.split('.');
     const moduleName = components[0];
 
@@ -1047,6 +1072,26 @@ export class Kernel {
     return fqnInfo;
   }
 
+  /**
+   * Determines whether the provided FQN corresponds to a valid, exported type
+   * from any currently loaded assembly.
+   *
+   * @param fqn the tested FQN.
+   *
+   * @returns `true` IIF the FQN corresponds to a know exported type.
+   */
+  #isVisibleType(fqn: spec.FQN): boolean {
+    try {
+      /* ignored */ this.#typeInfoForFqn(fqn);
+      return true;
+    } catch (e) {
+      if (e instanceof JsiiFault) {
+        return false;
+      }
+      throw e;
+    }
+  }
+
   #typeInfoForMethod(
     methodName: string,
     fqn: string,

packages/@jsii/kernel/src/link.ts

@@ -1,5 +1,19 @@
-import { copyFileSync, linkSync, mkdirSync, readdirSync, statSync } from 'fs';
-import { join } from 'path';
+import {
+  copyFileSync,
+  linkSync,
+  mkdirSync,
+  readdirSync,
+  statSync,
+  symlinkSync,
+} from 'fs';
+import { dirname, join } from 'path';
+
+/**
+ * If `node` is started with `--preserve-symlinks`, the module loaded will
+ * preserve symbolic links instead of resolving them, making it possible to
+ * symbolically link packages in place instead of fully copying them.
+ */
+const PRESERVE_SYMLINKS = process.execArgv.includes('--preserve-symlinks');
 
 /**
  * Creates directories containing hard links if possible, and falls back on
@@ -9,6 +23,12 @@ import { join } from 'path';
  * @param destination is the new file or directory to create.
  */
 export function link(existing: string, destination: string): void {
+  if (PRESERVE_SYMLINKS) {
+    mkdirSync(dirname(destination), { recursive: true });
+    symlinkSync(existing, destination);
+    return;
+  }
+
   const stat = statSync(existing);
   if (!stat.isDirectory()) {
     try {