node_modules linker refactor

internal/linker/README.md

@@ -1,25 +1,13 @@
-# node package linker
+# NodeJS Linker for Bazel Runfiles
 
-It's not obvious why a "linker" is needed in nodejs.
-After all, programs use dynamic lookups at runtime so we expect no need for static linking.
+`yarn_install` and `npm_install` do not place `node_modules` adjacent to `package.json`, and there
+is currently no rule to copy outputs into the correct location. To make up for this limitation a
+"linker" is required.
 
-However, in the monorepo case, you develop a package and also reference it by name in the same repo.
-This means you need a workflow like `npm link` to symlink the package from the `node_modules/name` directory to `packages/name` or wherever the sources live.
-[lerna] does a similar thing, but at a wider scale: it links together a bunch of packages using a descriptor file to understand how to map from the source tree to the runtime locations.
-
-Under Bazel, we have exactly this monorepo feature. But, we want users to have a better experience than lerna: they shouldn't need to run any tool other than `bazel test` or `bazel run` and they expect programs to work, even when they `require()` some local package from the monorepo.
-
-To make this seamless, we run a linker as a separate program inside the Bazel action, right before node.
-It does essentially the same job as Lerna: make sure there is a `$PWD/node_modules` tree and that all the semantics from Bazel (such as LinkablePackageInfo provider) are mapped to the node module resolution algorithm, so that the node runtime behaves the same way as if the packages had been installed from npm.
-
-Note that the behavior of the linker depends on whether the package to link was declared as:
-
-1. a runtime dependency of a binary run by Bazel, which we call "statically linked", and which is resolved from Bazel's Runfiles tree or manifest
-1. a dependency declared by a user of that binary, which we call "dynamically linked", and which is resolved from the execution root
-
-In the future the linker should also generate `package.json` files so that things like `main` and `typings` fields are present and reflect the Bazel semantics, so that we can entirely eliminate custom loading and pathmapping logic from binaries we execute.
-
-[lerna]: https://github.com/lerna/lerna
+A critical piece of this is linker setup is the script run as part of `nodejs_binary` and
+`nodejs_test` executables. This script creates a symlink adjacent to `package.json` pointing to
+`node_modules` in the external repository generated by `(yarn|npm)_install`. Care has been taken
+to ensure it is thread safe (across both instances of the same executable and a mix).
 
 # Developing
 

internal/linker/link_node_modules.bzl

@@ -101,33 +101,13 @@ def write_node_modules_manifest(ctx, extra_data = [], mnemonic = None, link_work
                 _debug(ctx.var, "Linking %s: %s" % (k, v))
                 mappings[k] = v
 
-    # Convert mappings to a module sets (modules per package package_path)
-    # {
-    #     "package_path": {
-    #             "package_name": "source_path",
-    #             ...
-    #     },
-    #     ...
-    # }
-    module_sets = {}
-    for k, v in mappings.items():
-        map_key_split = k.split(":")
-        package_name = map_key_split[0]
-        package_path = map_key_split[1] if len(map_key_split) > 1 else ""
-        source_path = v[1]
-        if package_path not in module_sets:
-            module_sets[package_path] = {}
-        module_sets[package_path][package_name] = source_path
-
     # Write the result to a file, and use the magic node option --bazel_node_modules_manifest
     # The launcher.sh will peel off this argument and pass it to the linker rather than the program.
     prefix = ctx.label.name
     if mnemonic != None:
         prefix += "_%s" % mnemonic
     modules_manifest = ctx.actions.declare_file("_%s.module_mappings.json" % prefix)
     content = {
-        "bin": ctx.bin_dir.path,
-        "module_sets": module_sets,
         "roots": node_modules_roots,
         "workspace": ctx.workspace_name,
     }