RFC: Generate dependency manifest on modular build

[cristiano-belloni]

Having a way of generating a "dependency manifest" when we build a package would be useful for teams to allow-list / block-list dependencies, do static analysis on them or generate import maps / rewrite imports to use as a basis for module federation and ESM CDN integration. This is an open request for comments to define how the dependency manifest should be defined and what should it contain. Here are some of the preliminary requirements identified so far:

• The dependency manifest should contain every dependency in a package, taking into account hoisted dependencies.
• The dependency manifest should be generated for any kind of modular package (currently apps, views and plain packages) at build time.
• The generation of the dependency manifest should be independent from the underlying bundler used to build the package (esbuild w/ flag vs webpack vs rollup should generate the same dependency manifest)

Update:

• Dependencies (hoisted and non-hoisted) will be reported in the package.json in the dist directory.
• bundledDependencies will also be populated

Open questions:

• Should Workspace dependencies be marked differently from external dependencies in the dependency manifest?
• Should the dependency manifest contain the version range of a dependency, or the currently installed dependency according to the lockfile? Probably the first one is more useful (and easier / quicker to get), but need feedback on this.

[LukeSheard]

Given there's already a dependency manifest built directly into the package.json format I would suggest that what we do here is change the way modular handles generating package.json files as part of the build process.

esbuild w/ flag vs webpack vs rollup should generate the same dependency manifest

I completely agree with this sentiment. The generation of the dependency manifest should be independent of the underlying build tool - although likely we'll need to write an API interface for consuming the build stat output to generate the interface needed for package.json files.

Should Workspace dependencies be marked differently from external dependencies in the dependency manifest?

Let's think about current state here - with the notion of an app build then dependencies are actually bundled, should they be included in the manifest?. With package based builds then they are marked as external.

I would suggest that we likely want to include the dependencies in the manifest, package.json I believe do have a field for bundledDependencies which we could likely utilised.

Should the dependency manifest contain the version range of a dependency

I was pondering this as I wrote this reply - realistically I think we can only analyse the range of dependency and trust SerVer behind the scenes, otherwise if we start pinning versions of dependencies then for library builds then we could end up duplicating dependencies inside consumers node_modules directories.

[cristiano-belloni]

Given there's already a dependency manifest built directly into the package.json format I would suggest that what we do here is change the way modular handles generating package.json files as part of the build process.

I agree, it seems a sensible solution. I see we currently generate package.json only if we're building views / plain packages, but not if we're building applications. Should we include the package.json in apps too? I agree apps are not strictly consumed (ie imported), but they can be still analysed by external tools.

we'll need to write an API interface for consuming the build stat output to generate the interface needed for package.json files.

Do we? I'm experimenting with ts-morph (which we already using for the rename command) in this strawman PR and reading the imports from the source code is quite fast (I think one order of magnitude faster than building / typechecking) and, more importantly, it's independent from the build process output. It's true that we need to build anyway, but being decoupled from the build process means that:

1. we don't have to maintain the scraping of the build stats which might be incompatible btw different versions (webpack 4 / 5?) or write plugins (esbuild)
2. if we change the build process of views / plain packages, we don't have to wait for that task to be done to have dependency manifests outside of applications
3. we could theoretically offer dependency extraction as a separate command
4. we can modify the underlying build mechanism / bundler without changing the dependency manifest generation code
5. if, in the future, we want to give the option to rewrite external imports to leverage ESM module federation, this solution would work anyway and could be re-used to pass dependencies to the rewriting plugin before the build. It's true that we can rewrite dependencies on the fly, but generating an importMap before building is handy.
6. the code is much more compact

I was pondering this as I wrote this reply - realistically I think we can only analyse the range of dependency and trust SerVer behind the scenes, otherwise if we start pinning versions of dependencies then for library builds then we could end up duplicating dependencies inside consumers node_modules directories.

I agree. Technically it's possible to leverage yarn why programatically to extract currently installed dependencies without parsing the lockfile, but I think for all intents and purposes it's easier, faster and more correct to just include the ranges in the manifest.

[LukeSheard]

Seems like we're aligned here - I'm not concerned about the technical implementation as so far as it's correct. So if we're already using ts-morph elsewhere then we can continue using that.