Reduce size of module dependency graph #1342

Krinkle · 2018-12-22T20:59:15Z

The qunit npm package has quite a lot of dependencies installed when used in another project.

The total number of packages installed according to (as of QUnit 2.8.0) is 143 packages. (Not including dev dependencies, that are only installed for QUnit contributors.)

Graph visualisation: https://npm.anvaka.com/#/view/2d/qunit/2.8.0
Tree visualisation: https://snyk.io/test/npm/qunit/2.8.0?tab=dependencies

Assuming we won't be investing in a way to sandbox QUnit, I believe this is beyond what we can reasonably expect a developer to audit before installing. Keeping in mind that merely running npm install in a Git project using qunit, is enough for any of these packages turned bad to start scraping process.env, ~/.ssh, etc (e.g. through postinstall hooks, or at run-time).

I'm aiming with this task to take a serious look at what abstractions we need and are providing value, and which we might be better off not using a package for. For problems where the abstractions can be "simple enough to obviously have no problems", it might be better to simply inline it within our repo (with proper licensing information inline, if based on an existing package). In such case, I believe the theoretical value gained from future updates is non-existent or small enough to not outweigh the loss in understandability and security of QUnit overall. If in these areas we do find a rare bug one day, we can easily update it by hand.

Goal: Maximum 20, preferably less than 10.

QUnit prides simplicity, I believe. We've done well to uphold that in our API. I'm hoping we can reach simplicity in our dependency graph as well!

Relating reading:

And also:

The text was updated successfully, but these errors were encountered:

trentmwillis · 2018-12-23T18:55:58Z

I'm on board with this!

Package 'findup-sync': * Tree: (96 packages) |- detect-file@1.0.0 |- is-glob@3.1.0 \ ... |- micromatch@3.1.10 \ ... |-resolve-dir@1.0.1 \ ... * Summary: We used it in one place only, for finding a 'package.json' file. The findup-async library seems to only offer features we don't need. For example: - 'detect-file' provides a case-insensitive version of `fs.stat`. - 'is-glob' and 'micromatch' helps provide a pattern-based search. - 'resolve-dir' allows the start directory to contain tilde (~) for `$HOME` and `@` for package-bound traversal. For our use case, we only needed a loop that calls the built-in `require('fs').stat()` and `require('path').dirname()`. On second thought, I realised that we can avoid this entire problem by simply using `require('<module>/package.json')`. Which allows us to directly include a file from the module directory. Package 'exists-stat' * (1 package, no sub-dependencies.) * Summary: We used it in one place only. Added the function to cli/utils directly. Package 'walk-sync' * Tree: (7 packages) |- ensure-posix-path@1.0.2 |- matcher-collection@1.0.5 |- minimatch@3.0.4 |- brace-expansion@1.1.11 |- balanced-match@1.0.0 |- concat-map@0.0.1 * Summary: We used in two places. I created a simplified version (directly based on Minimatch) in cli/utils and updated callers. Ref #1342.

Similar to the previous commit, I considered a simple approach of our own based on the built-in Node.js API. The fs.watch() API is actually quite good in Node 6+ (certainly much better than it used to be). But, there are two notable issues that I think we should care about: 1. Its "recursive" feature is lacking on Linux (only stable on macOS and Windows). 2. Its ability to distinguish between create, update and remove events isn't very good. This last point would be fairly easily to do on the consumer side with a quick fs.stat() call. We could even omit it entirely given we only use it for one word (a verb) in the CLI output. But the first point (recursion) is slightly more involved than I'd like to maintain locally. All the reviewed packages essentially handle this the same way. They create a non-recursive fs.watch() for each sub directory found on the system, track them in an object. Then, start new ones as needed when new directories are created, and stop old one when directories are removed. That's about 100 lines of simple code. Where they differ is: * How many extra features they provide. * How many simple functions for non-critical code are delegated to other packages. * Whether they use native "recursive" when available (on macOS/Windows). privide in addition to that I'm proposing we go with node-watch. This package is well-maintained, tested with the latest Node versions, optimised for Node 6+, and provides no additional features, and is dependency-free. > sane@4.0.2 (current) > Dependencies: ⚠️ > 119 packages. > (concerning in terms of dicipline and security) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 25 commits, 8 contributors, 2 major releases. > Modern: *️⃣ > Requires Node 6+, but not yet tested on Node 10. > gaze@1.1.3 > Dependencies: *️⃣ > 14 packages. > (okay, but could be better.) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 6 contributors, 1 release. > Modern: ✅ > Requires Node 4+, tested with Node 10. > watch@1.0.2: > Dependencies: ✅ > 3 packages. > License: ✅ > Apache-2.0. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Still supports Node 0.1. No visible CI or commit activity > indicating testing with recent Node releases. > node-watch@0.5.9: > Dependencies: ✅ > 1 package (dependency-free). > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 3 contributors, 3 minor releases. > Modern: ✅ > Requires Node 6+, tested with Node 10. > Uses native fs.watch/recursive support where available. > filewatcher@3.0.1: > Dependencies: ✅ > 2 packages (1 dependency). > License: ✅ > MIT. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Last tested with Node 0.10. > Uses per-file watching and polling for all platforms. Fixes #1342.

Package 'findup-sync': * Tree: (96 packages) |- detect-file@1.0.0 |- is-glob@3.1.0 \ ... |- micromatch@3.1.10 \ ... |-resolve-dir@1.0.1 \ ... * Summary: We used it in one place only, for finding a 'package.json' file. The findup-async library seems to only offer features we don't need. For example: - 'detect-file' provides a case-insensitive version of `fs.stat`. - 'is-glob' and 'micromatch' helps provide a pattern-based search. - 'resolve-dir' allows the start directory to contain tilde (~) for `$HOME` and `@` for package-bound traversal. For our use case, we only needed a loop that calls the built-in `require('fs').stat()` and `require('path').dirname()`. On second thought, I realised that we can avoid this entire problem by simply using `require('<module>/package.json')`. Which allows us to directly include a file from the module directory. Package 'exists-stat' * (1 package, no sub-dependencies.) * Summary: We used it in one place only. Added the function to cli/utils directly. Package 'walk-sync' * Tree: (7 packages) |- ensure-posix-path@1.0.2 |- matcher-collection@1.0.5 |- minimatch@3.0.4 |- brace-expansion@1.1.11 |- balanced-match@1.0.0 |- concat-map@0.0.1 * Summary: We used in two places. I created a simplified version (directly based on Minimatch) in cli/utils and updated callers. Ref #1342.

Similar to the previous commit, I considered a simple approach of our own based on the built-in Node.js API. The fs.watch() API is actually quite good in Node 6+ (certainly much better than it used to be). But, there are two notable issues that I think we should care about: 1. Its "recursive" feature is lacking on Linux (only stable on macOS and Windows). 2. Its ability to distinguish between create, update and remove events isn't very good. This last point would be fairly easily to do on the consumer side with a quick fs.stat() call. We could even omit it entirely given we only use it for one word (a verb) in the CLI output. But the first point (recursion) is slightly more involved than I'd like to maintain locally. All the reviewed packages essentially handle this the same way. They create a non-recursive fs.watch() for each sub directory found on the system, track them in an object. Then, start new ones as needed when new directories are created, and stop old one when directories are removed. That's about 100 lines of simple code. Where they differ is: * How many extra features they provide. * How many simple functions for non-critical code are delegated to other packages. * Whether they use native "recursive" when available (on macOS/Windows). privide in addition to that I'm proposing we go with node-watch. This package is well-maintained, tested with the latest Node versions, optimised for Node 6+, and provides no additional features, and is dependency-free. > sane@4.0.2 (current) > Dependencies: ⚠️ > 119 packages. > (concerning in terms of dicipline and security) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 25 commits, 8 contributors, 2 major releases. > Modern: *️⃣ > Requires Node 6+, but not yet tested on Node 10. > gaze@1.1.3 > Dependencies: *️⃣ > 14 packages. > (okay, but could be better.) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 6 contributors, 1 release. > Modern: ✅ > Requires Node 4+, tested with Node 10. > watch@1.0.2: > Dependencies: ✅ > 3 packages. > License: ✅ > Apache-2.0. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Still supports Node 0.1. No visible CI or commit activity > indicating testing with recent Node releases. > node-watch@0.5.9: > Dependencies: ✅ > 1 package (dependency-free). > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 3 contributors, 3 minor releases. > Modern: ✅ > Requires Node 6+, tested with Node 10. > Uses native fs.watch/recursive support where available. > filewatcher@3.0.1: > Dependencies: ✅ > 2 packages (1 dependency). > License: ✅ > MIT. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Last tested with Node 0.10. > Uses per-file watching and polling for all platforms. Fixes #1342.

Similar to the previous commit, I considered a simple approach of our own based on the built-in Node.js API. The fs.watch() API is actually quite good in Node 6+ (certainly much better than it used to be). But, there are two notable issues that I think we should care about: 1. Its "recursive" feature is lacking on Linux (only stable on macOS and Windows). 2. Its ability to distinguish between create, update and remove events isn't very good. This last point would be fairly easily to do on the consumer side with a quick fs.stat() call. We could even omit it entirely given we only use it for one word (a verb) in the CLI output. But the first point (recursion) is slightly more involved than I'd like to maintain locally. All the reviewed packages essentially handle this the same way. They create a non-recursive fs.watch() for each sub directory found on the system, track them in an object. Then, start new ones as needed when new directories are created, and stop old one when directories are removed. That's about 100 lines of simple code. Where they differ is: * How many extra features they provide. * How many simple functions for non-critical code are delegated to other packages. * Whether they use native "recursive" when available (on macOS/Windows). privide in addition to that I'm proposing we go with node-watch. This package is well-maintained, tested with the latest Node versions, optimised for Node 6+, and provides no additional features, and is dependency-free. > sane@4.0.2 (current) > Dependencies: ⚠️ > 119 packages. > (concerning in terms of dicipline and security) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 25 commits, 8 contributors, 2 major releases. > Modern: *️⃣ > Requires Node 6+, but not yet tested on Node 10. > gaze@1.1.3 > Dependencies: *️⃣ > 14 packages. > (okay, but could be better.) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 6 contributors, 1 release. > Modern: ✅ > Requires Node 4+, tested with Node 10. > watch@1.0.2: > Dependencies: ✅ > 3 packages. > License: ✅ > Apache-2.0. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Still supports Node 0.1. No visible CI or commit activity > indicating testing with recent Node releases. > node-watch@0.5.9: > Dependencies: ✅ > 1 package (dependency-free). > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 3 contributors, 3 minor releases. > Modern: ✅ > Requires Node 6+, tested with Node 10. > Uses native fs.watch/recursive support where available. > filewatcher@3.0.1: > Dependencies: ✅ > 2 packages (1 dependency). > License: ✅ > MIT. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Last tested with Node 0.10. > Uses per-file watching and polling for all platforms. Fixes qunitjs#1342.

Similar to the previous commit, I considered a simple approach of our own based on the built-in Node.js API. The fs.watch() API is actually quite good in Node 6+ (certainly much better than it used to be). But, there are two notable issues that I think we should care about: 1. Its "recursive" feature is lacking on Linux (only stable on macOS and Windows). 2. Its ability to distinguish between create, update and remove events isn't very good. This last point would be fairly easily to do on the consumer side with a quick fs.stat() call. We could even omit it entirely given we only use it for one word (a verb) in the CLI output. But the first point (recursion) is slightly more involved than I'd like to maintain locally. All the reviewed packages essentially handle this the same way. They create a non-recursive fs.watch() for each sub directory found on the system, track them in an object. Then, start new ones as needed when new directories are created, and stop old one when directories are removed. That's about 100 lines of simple code. Where they differ is: * How many extra features they provide. * How many simple functions for non-critical code are delegated to other packages. * Whether they use native "recursive" when available (on macOS/Windows). privide in addition to that I'm proposing we go with node-watch. This package is well-maintained, tested with the latest Node versions, optimised for Node 6+, and provides no additional features, and is dependency-free. > sane@4.0.2 (current) > Dependencies: ⚠️ > 119 packages. > (concerning in terms of dicipline and security) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 25 commits, 8 contributors, 2 major releases. > Modern: *️⃣ > Requires Node 6+, but not yet tested on Node 10. > gaze@1.1.3 > Dependencies: *️⃣ > 14 packages. > (okay, but could be better.) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 6 contributors, 1 release. > Modern: ✅ > Requires Node 4+, tested with Node 10. > watch@1.0.2: > Dependencies: ✅ > 3 packages. > License: ✅ > Apache-2.0. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Still supports Node 0.1. No visible CI or commit activity > indicating testing with recent Node releases. > node-watch@0.5.9: > Dependencies: ✅ > 1 package (dependency-free). > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 3 contributors, 3 minor releases. > Modern: ✅ > Requires Node 6+, tested with Node 10. > Uses native fs.watch/recursive support where available. > filewatcher@3.0.1: > Dependencies: ✅ > 2 packages (1 dependency). > License: ✅ > MIT. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Last tested with Node 0.10. > Uses per-file watching and polling for all platforms. Fixes #1342.

Package 'findup-sync': * Tree: (96 packages) |- detect-file@1.0.0 |- is-glob@3.1.0 \ ... |- micromatch@3.1.10 \ ... |-resolve-dir@1.0.1 \ ... * Summary: We used it in one place only, for finding a 'package.json' file. The findup-async library seems to only offer features we don't need. For example: - 'detect-file' provides a case-insensitive version of `fs.stat`. - 'is-glob' and 'micromatch' helps provide a pattern-based search. - 'resolve-dir' allows the start directory to contain tilde (~) for `$HOME` and `@` for package-bound traversal. For our use case, we only needed a loop that calls the built-in `require('fs').stat()` and `require('path').dirname()`. On second thought, I realised that we can avoid this entire problem by simply using `require('<module>/package.json')`. Which allows us to directly include a file from the module directory. Package 'exists-stat' * (1 package, no sub-dependencies.) * Summary: We used it in one place only. Added the function to cli/utils directly. Package 'walk-sync' * Tree: (7 packages) |- ensure-posix-path@1.0.2 |- matcher-collection@1.0.5 |- minimatch@3.0.4 |- brace-expansion@1.1.11 |- balanced-match@1.0.0 |- concat-map@0.0.1 * Summary: We used in two places. I created a simplified version (directly based on Minimatch) in cli/utils and updated callers. Ref #1342.

Similar to the previous commit, I considered a simple approach of our own based on the built-in Node.js API. The fs.watch() API is actually quite good in Node 6+ (certainly much better than it used to be). But, there are two notable issues that I think we should care about: 1. Its "recursive" feature is lacking on Linux (only stable on macOS and Windows). 2. Its ability to distinguish between create, update and remove events isn't very good. This last point would be fairly easily to do on the consumer side with a quick fs.stat() call. We could even omit it entirely given we only use it for one word (a verb) in the CLI output. But the first point (recursion) is slightly more involved than I'd like to maintain locally. All the reviewed packages essentially handle this the same way. They create a non-recursive fs.watch() for each sub directory found on the system, track them in an object. Then, start new ones as needed when new directories are created, and stop old one when directories are removed. That's about 100 lines of simple code. Where they differ is: * How many extra features they provide. * How many simple functions for non-critical code are delegated to other packages. * Whether they use native "recursive" when available (on macOS/Windows). privide in addition to that I'm proposing we go with node-watch. This package is well-maintained, tested with the latest Node versions, optimised for Node 6+, and provides no additional features, and is dependency-free. > sane@4.0.2 (current) > Dependencies: ⚠️ > 119 packages. > (concerning in terms of dicipline and security) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 25 commits, 8 contributors, 2 major releases. > Modern: *️⃣ > Requires Node 6+, but not yet tested on Node 10. > gaze@1.1.3 > Dependencies: *️⃣ > 14 packages. > (okay, but could be better.) > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 6 contributors, 1 release. > Modern: ✅ > Requires Node 4+, tested with Node 10. > watch@1.0.2: > Dependencies: ✅ > 3 packages. > License: ✅ > Apache-2.0. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Still supports Node 0.1. No visible CI or commit activity > indicating testing with recent Node releases. > node-watch@0.5.9: > Dependencies: ✅ > 1 package (dependency-free). > License: ✅ > MIT. > Supported: ✅ > 2018 saw 22 commits, 3 contributors, 3 minor releases. > Modern: ✅ > Requires Node 6+, tested with Node 10. > Uses native fs.watch/recursive support where available. > filewatcher@3.0.1: > Dependencies: ✅ > 2 packages (1 dependency). > License: ✅ > MIT. > Supported: ⚠️ > 2018 saw no commits or releases. > Modern: ⚠️ > Last tested with Node 0.10. > Uses per-file watching and polling for all platforms. Fixes #1342.

Krinkle · 2018-12-29T02:09:23Z

$ npm install --prod   
...
added 9 packages
found 0 vulnerabilities

$ npm ls
qunit@2.8.1-pre
+-- commander@2.12.2
+-- js-reporters@1.2.1
+-- minimatch@3.0.4
| `-- brace-expansion@1.1.11
|   +-- balanced-match@1.0.0
|   `-- concat-map@0.0.1
+-- node-watch@0.5.9
`-- resolve@1.5.0
  `-- path-parse@1.0.6

Before:
QUnit 2.8.0 has 142 dependencies.

After:
QUnit 2.9.1 has 9 dependencies.

Before	After

Krinkle added Type: Meta Seek input from maintainers and contributors. Component: CLI Type: Bug Something isn't working right. labels Dec 22, 2018

Krinkle self-assigned this Dec 27, 2018

Krinkle mentioned this issue Dec 28, 2018

CLI: Reduce the size of the dependencies tree #1345

Merged

Krinkle closed this as completed in #1345 Dec 28, 2018

Krinkle added this to the 2.9 milestone Jan 3, 2019

Krinkle mentioned this issue Oct 11, 2019

Reduce number of Puppeteer installer dependencies puppeteer/puppeteer#5026

Closed

This was referenced Aug 16, 2020

Core: Drop support for Node 6 and Node 8, require Node 10+ #1464

Merged

Reduce size of dependency graph 11ty/eleventy#1375

Closed

Krinkle mentioned this issue Aug 26, 2020

Revisit the watch option and chokidar dependency mozilla/nunjucks#1227

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce size of module dependency graph #1342

Reduce size of module dependency graph #1342

Krinkle commented Dec 22, 2018 •

edited

Loading

trentmwillis commented Dec 23, 2018

Krinkle commented Dec 29, 2018 •

edited

Loading

Reduce size of module dependency graph #1342

Reduce size of module dependency graph #1342

Comments

Krinkle commented Dec 22, 2018 • edited Loading

trentmwillis commented Dec 23, 2018

Krinkle commented Dec 29, 2018 • edited Loading

Krinkle commented Dec 22, 2018 •

edited

Loading

Krinkle commented Dec 29, 2018 •

edited

Loading