[RRFC] Add top level binDependencies to package.json

[Raynos]

Motivation ("The Why")

When looking at the output of npm ls, reviewing package-lock.json or opening node_modules in my text editor I would like to only see dependencies that I require directly in my library or application.

If I have a small library or application with 3-10 dependencies and nyc, then 95% of node_modules is transitive dependencies of nyc instead of code that I am actually using in my application / library.

When building an application or a library I want to use certain dev dependencies that are only used as binaries in the scripts section of my package.json ; These dev dependencies are never used by a require statement.

Examples: typescript, tslint, standard, nyc, electron, webpack, testcafe, browserify, etc.

A long time ago, we used to install these dependencies globally before npm added ./node_modules/.bin to $PATH in the scripts section. An application developer was expected to have a copy of nyc and eslint installed with npm i nyc eslint -g.

Example

I have a library that uses aws-sdk & tape for testing, But also nyc & tslint for as binaries. The output of npm ls and ls node_modules shows that there are 450 dependencies.

If I move all the binaries to be globally installed npm ls and ls node_modules show that there are only 25 dependencies that are actually used by my library.

It is frustrating for me to have to look at dependencies in node_modules that are not relevant to my library. I am willing to trust these binaries like nyc and tslint as black boxes.

In other communities, like the go community, these binary only dependencies are installed as a single binary and are generally fetched in the Makefile. These binary only dependencies do not exist in the vendor directory since they are treated as single standalone binary blackboxes.

How

Current Behaviour

Npm installs binary dependencies into node_modules and I add them to devDependencies

Desired Behaviour

Npm installs binary dependendencies somewhere else that is not ./node_modules ( Could be in $HOME/.npm/... ) and I add them to binDependencies

References

I have implemented a prototype in userland https://github.com/Raynos/npm-bin-deps

This userland prototype is missing some features like

• npm install tslint -B ; I have to hand edit the binDependencies declaration in my package.json
• Adding the bin dependencies to $PATH in scripts ; I currently run npr standard instead of standard in scripts.

Caveats

Is tap devDependencies or binDependencies ?

tap is both a library you require and a binary dependency. It's not a good fit for binDependencies.

Since a lot of npm users ( installers ) are not going to understand the nuance of this. One possible approach would be for the author of tap & nyc , etc to decide if their library is a dev dependency of a bin dependency. This could be a new field in package.json like isBinOnly: true ;

Then the npm client when running npm install nyc --save-bin would know that it is isBinOnly and add it to binDependencies. Running npm install tap --save-bin would know that it is not isBinOnly and would add it to devDependencies and warn the user that tap cannot be a binDependencies.

Adding the isBinOnly field actually makes the entire binDependencies functionality opt in based on package publishers.

Reproducable tests in CI

By design of treating these binary only dev dependencies as a blackbox they do not show up in package-lock.json ; This means I do not have to review hundreds of lines of changes when they get updated but I also do not have reproducible builds for testing since nested dependencies could change on re-install.

For reproducible builds it might be useful to have an optional bin-package-lock.json ; By having the binDependencies write into their own lock file I can easily chose to not review this file in code review since I know that these dependencies are not part of my app / library / ./node_modules

The reason for a seperate lock file would be because the current package-lock.json is for ./node_modules and by design binDependencies are not installed in node_modules but instead in some other location like $HOME/.npm .

Challenging to do printf debugging on binary dependencies

As a user of npm-bin-deps there have been a couple of times I wanted to fix a bug in a binary dependency and I had to manually open ~/.config/npm-bin-deps/... to do printf style debugging.

This is a pretty rare use case and the best recommendation I have is to manually re-install the dependency as a dev dependency, do your printf debugging / bugfixes / forking / pull request and re-install as a bin dependency once stable.

[Raynos]

I have implemented this in userland https://github.com/Raynos/npm-bin-deps

I can open an RFC if there is interest in this, but i would not have the capacity to contribute this feature to the npm cli.

I am happy to join the Open RFC meeting to discuss and present this idea.

[Raynos]

Reference to Open RFC meeting : https://www.youtube.com/watch?v=43JFgdJzakI&t=13m30s

[Raynos]

ls node_modules ; I don't think people do it that much.

Running ls node_modules is very uncommon. What is far more common is expanding the node_modules directory in Visual Studio Code to read the source code of a dependency or to add a console.log statement in a dependency.

I used ls node_modules as an example but i really meant exploring the file system in my Text Editor.

Some bin dependencies want to same library as the binary, for example tap or jest.

This is a really good point, I don't personally use a binary that I also use a library you can require ; I think installing tap as a binDependencies is a bug. That would be a bad idea.

npm ls --prod can be used. npm ls --depth can be used.

These help with npm ls and that's great 👍

These do not help with exploring ./node_modules in my text editor to read the source code, Isaacs mentioned that example of I have one dependency and I have tap ; when I want to read the source code of that one dependency I have to scroll through a huge list of things I do not care about.

Also the package-lock.json would be smaller and easier to review in code review. That's probably the biggest concrete win I have with the user land npm-bin-deps is the ability to git add -p package-lock.json and actually make sense of it.

cc @isaacs

[Raynos]

The open RFC meeting mentioned unlisted dependencies.

I know pnpm very specifically breaks code that relies on unlisted dependencies. They have pnpmfile.js

https://pnpm.js.org/en/pnpmfile#hooks ; I have used the pnpm hook system to "fix" a nested dependency and to rewrite its package.json to actually depend on its unlisted dependencies, this would unblock me from waiting for the transient dependency to fix itself.

Moving various transient dependencies into ./node_modules/node_modules would solve the explore ./node_modules in VS code problem. It would be far easier to find my direct dependencies that way.

Moving transient dependencies into ./node_modules/node_modules does not make package-lock.json easier to review.

[Raynos]

Some bin dependencies want to same library as the binary, for example tap or jest.

A user of tap probably does not understand the nuance of whether any given dev dependency is a dev dependency or a bin dependency.

Based on this use case I am convinced that the author of tap and eslint etc should opt into "installAsBinDependency" in their package.json when they publish their package to NPM.