yarn vs npm install node_modules bloat

[zuidell]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?
Yarn is bloating the node_modules directory, 5x file count vs npm and 2.5x file size

If the current behavior is a bug, please provide the steps to reproduce.

Using create-react-app as an example

1. create-react-app exampleApp
2. ~110,000 files in ./exampleApp, ~180 MB
3. rm -r ./exampleApp/node_modules
4. yarn (similar results)
5. rm -r ./exampleApp/node_modules
6. npm install (instead of using yarn)
7. ~21,000 files in ./exampleApp, ~80 MB

Folder Example:
node_modules (after npm install, 24 KB)
-> babel-helpers
-> -> lib
-> -> package.json
-> -> README.md

node_modules (after yarn install, 2 MB, everything is the same except sub node_modules)
-> babel-helpers
-> -> lib
-> -> node_modules
-> -> package.json
-> -> README.md

What is the expected behavior?
I wouldn't expect a yarn install to have 5x the files of an NPM install on the same package list.

Please mention your node.js, yarn and operating system version.
node - 7.5.0
yarn - 0.19.1
macOS Sierra Version 10.12.3 (verified also happening on Windows 7)

[rally25rs]

For an application that I have here to test with:

NPM version: 3.10.3
Yarn version: 0.19.1

after npm install:

217,615,400 bytes (350.6 MB on disk) for 45,462 items

after yarn install:

224,005,232 bytes (391.5 MB on disk) for 54,394 items

I started trying to compare the 2 node_modules folders to see what is different, and it looks like yarn and npm have different ways of determining what to do when multiple versions of a module are needed. It looks like Yarn duplicates more under submodules node_modules, and npm either gets lucky, or determines which module version has more dependencies and puts that version at the root, resulting in less sub-module duplication.

In other words, if multiple versions of a module are needed, NPM and Yarn seem to put different versions at the root level, and then submodules that don't use that version at the root level get their own copy under the submodule's node_modules.

So conceptually I end up with something like:

If modules A, B and C all depend on module X.
A depends on X v1.
B and C depend on X v2.

npm:

A
 X v1
B
  (uses X v2 from root)
C
  (uses X v2 from root)
X v2

Yarn:

A
  (uses X v1 from root)
B
  X v2
C
  X v2
X v1

so Yarn ends up with more module duplication...

I have no idea if this is just a coincidence based on the order that modules are installed, or a product of the algorithm that flattens the modules... someone who knows the Yarn code may be able to investigate further...

---

Just a quick update post-fix for this issue:

after yarn install with v19.1 (before the fix):

224,005,232 bytes (391.5 MB on disk) for 54,394 items

after yarn install with v21.3 (after the fix):

163,226,945 bytes (268.7 MB on disk) for 35,074 items

👏

[bestander]

should be fixed by #2676