Setup multiple local modules with npm link + dedupe

[emmenko]

Hi everybody,

I'm trying to use npm link to develop multiple modules locally but it's not working as expected.
I'm not sure if it's actually possible with this setup or if I'm doing something wrong.
Hopefully someone can shred some light here ;)

Let's say I have this kind of structure.

.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
└── package.json

The main app package.json has dependencies of all three modules

{
  "name": "app",
  "version": "0.0.0",
  "main": "./app/main.js",
  "dependencies": {
    "a": "file:./modules/a",
    "b": "file:./modules/b",
    "c": "file:./modules/c"
  }
}

The module a has underscore as only dependencies.

The module b has module a as dependency.

The module c has also module a as dependency.

// module `a`
{
  "name": "a",
  "version": "0.0.0",
  "main": "./main.js",
  "dependencies": {
    "underscore": "1.8.x"
  }
}

// module `b`
{
  "name": "b",
  "version": "0.0.0",
  "main": "./main.js",
  "dependencies": {
    "a": "file:../a"
  }
}

// module `c`
{
  "name": "c",
  "version": "0.0.0",
  "main": "./main.js",
  "dependencies": {
    "a": "file:../a"
  }
}

Scenarios

The easiest way

So if I install the app dependencies I get this

$ npm i
c@0.0.0 node_modules/c

b@0.0.0 node_modules/b

a@0.0.0 node_modules/a
└── underscore@1.8.2

$ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
├── node_modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
└── package.json

Now if I change something in one of the modules I have to re-install them. Running npm i doesn't do anything of course since npm doesn't see any change in the modules definition (e.g.: version bump).

So one thing we can do is rm -rf node_modules && npm i. This will always work but it may be slow depending on your dependency tree.

Funny thing, running npm i -f has a different output.

$ npm i -f
npm WARN using --force I sure hope you know what you are doing.

a@0.0.0 node_modules/a
└── underscore@1.8.2

b@0.0.0 node_modules/b
└── a@0.0.0 (underscore@1.8.2)

c@0.0.0 node_modules/c
└── a@0.0.0 (underscore@1.8.2)
emmenko: ~/dev/src/npm-link-test $ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   └── package.json
│   └── c
│       ├── main.js
│       └── package.json
├── node_modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── a
│   │   │       ├── main.js
│   │   │       ├── node_modules
│   │   │       │   └── underscore
│   │   │       │       ├── LICENSE
│   │   │       │       ├── README.md
│   │   │       │       ├── package.json
│   │   │       │       ├── underscore-min.js
│   │   │       │       ├── underscore-min.map
│   │   │       │       └── underscore.js
│   │   │       └── package.json
│   │   └── package.json
│   └── c
│       ├── main.js
│       ├── node_modules
│       │   └── a
│       │       ├── main.js
│       │       ├── node_modules
│       │       │   └── underscore
│       │       │       ├── LICENSE
│       │       │       ├── README.md
│       │       │       ├── package.json
│       │       │       ├── underscore-min.js
│       │       │       ├── underscore-min.map
│       │       │       └── underscore.js
│       │       └── package.json
│       └── package.json
└── package.json

You can see the nested (duplicate) dependencies in each module.

The symlink way

So npm has the ability to symlink a package folder which is particularly convenient for developing modules locally.
Let's try it then (I will remove first node_modules to have a clean start).

$ cd modules/a
$ npm link
underscore@1.8.2 node_modules/underscore
/usr/local/lib/node_modules/a -> /Users/emmenko/dev/src/npm-link-test/modules/a
$ cd ..
$ cd b
$ npm link
a@0.0.0 node_modules/a
└── underscore@1.8.2
/usr/local/lib/node_modules/b -> /Users/emmenko/dev/src/npm-link-test/modules/b
$ cd ..
$ cd c
$ npm link
a@0.0.0 node_modules/a
└── underscore@1.8.2
/usr/local/lib/node_modules/c -> /Users/emmenko/dev/src/npm-link-test/modules/c
$ cd ..
$ cd ..
$ npm link a
/Users/emmenko/dev/src/npm-link-test/node_modules/a -> /usr/local/lib/node_modules/a -> /Users/emmenko/dev/src/npm-link-test/modules/a
$ npm link b
/Users/emmenko/dev/src/npm-link-test/node_modules/b -> /usr/local/lib/node_modules/b -> /Users/emmenko/dev/src/npm-link-test/modules/b
$ npm link c
/Users/emmenko/dev/src/npm-link-test/node_modules/c -> /usr/local/lib/node_modules/c -> /Users/emmenko/dev/src/npm-link-test/modules/c
$ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── a
│   │   │       ├── main.js
│   │   │       ├── node_modules
│   │   │       │   └── underscore
│   │   │       │       ├── LICENSE
│   │   │       │       ├── README.md
│   │   │       │       ├── package.json
│   │   │       │       ├── underscore-min.js
│   │   │       │       ├── underscore-min.map
│   │   │       │       └── underscore.js
│   │   │       └── package.json
│   │   └── package.json
│   └── c
│       ├── main.js
│       ├── node_modules
│       │   └── a
│       │       ├── main.js
│       │       ├── node_modules
│       │       │   └── underscore
│       │       │       ├── LICENSE
│       │       │       ├── README.md
│       │       │       ├── package.json
│       │       │       ├── underscore-min.js
│       │       │       ├── underscore-min.map
│       │       │       └── underscore.js
│       │       └── package.json
│       └── package.json
├── node_modules
│   ├── a -> /usr/local/lib/node_modules/a
│   ├── b -> /usr/local/lib/node_modules/b
│   └── c -> /usr/local/lib/node_modules/c
└── package.json

So we can see the symlinks in node_modules, but we also see that each module has duplicated dependencies.

What we actually want is a to be a top level dependency (since is shared between modules) and most importantly that underscore is not duplicated between the other modules.

Again, npm (should) come to the rescue as it provides a way to reduce duplication via the dedupe command.

Searches the local package tree and attempts to simplify the overall structure by moving dependencies further up the tree, where they can be more effectively shared by multiple dependent packages.

Let's run it then and see what happens:

$ npm dedupe
$ tree
.
├── app
│   └── main.js
├── modules
│   ├── a
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── underscore
│   │   │       ├── LICENSE
│   │   │       ├── README.md
│   │   │       ├── package.json
│   │   │       ├── underscore-min.js
│   │   │       ├── underscore-min.map
│   │   │       └── underscore.js
│   │   └── package.json
│   ├── b
│   │   ├── main.js
│   │   ├── node_modules
│   │   │   └── a
│   │   │       ├── main.js
│   │   │       ├── node_modules
│   │   │       │   └── underscore
│   │   │       │       ├── LICENSE
│   │   │       │       ├── README.md
│   │   │       │       ├── package.json
│   │   │       │       ├── underscore-min.js
│   │   │       │       ├── underscore-min.map
│   │   │       │       └── underscore.js
│   │   │       └── package.json
│   │   └── package.json
│   └── c
│       ├── main.js
│       ├── node_modules
│       │   └── a
│       │       ├── main.js
│       │       ├── node_modules
│       │       │   └── underscore
│       │       │       ├── LICENSE
│       │       │       ├── README.md
│       │       │       ├── package.json
│       │       │       ├── underscore-min.js
│       │       │       ├── underscore-min.map
│       │       │       └── underscore.js
│       │       └── package.json
│       └── package.json
├── node_modules
│   ├── a -> /usr/local/lib/node_modules/a
│   ├── b -> /usr/local/lib/node_modules/b
│   └── c -> /usr/local/lib/node_modules/c
└── package.json

Hmm, nothing changed.

So is this how it is suppose to work? Does it actually work with local paths and symlinks?

$ npm -v
2.7.3

[smikes]

Since I only use npm link during development, I would tend to approach the problem like this:

Assuming a directory with the checked-out repositories like this:

$ ls 
a      app      b      c

First set up global linked versions of a,b,c

$ (cd a && npm link)
$ (cd b && npm link)
$ (cd c && npm link)

link a to c,b

$ (cd b && npm link a)
$ (cd c && npm link a)

Then add a,b,c dependencies as links

$ (cd app && npm link a b c)

Now - don't look at the tree. It doesn't matter how often you see underscore in the linked tree, because there's really only one version, the one installed under a\node_modules.

If you need to prove it to yourself, then go to a scratch directory, and npm install app, and see that there's only one a and one underscore.

Of course there is another approach, which is just to bump all your packages to version 1.0.0 and then increment your versions when republishing. This way you make the version dependencies explicit.

[ehsalazar]

We haven’t heard back and we’re trying to clean up some old issues.  If this is still a problem, can you please reply and let us know?  We’ll be happy to reopen if necessary.

[hon2a]

I just came across this problem as well and with a scenario that is not easily remedied by the suggested workarounds.

I have 2 modules: lib and app. They both have React as dependency and module app also depends on lib. If I install lib into app using npm install, it works all right, because the top-most React is used in both app and lib files on build (building from app, the lib is just sources). If I don't want to even have the two React installations in my tree, I can just run npm dedupe.

However, since I'm developing both modules side by side, it'd really help me to use npm link instead of hard-installing the modules and then needing to version-bump + update or uninstall + reinstall them on every small change. But when I use npm link, the lib module isn't really inside app folder, so npm dedupe doesn't unify the React dependencies and when building the app using browserify, the app and lib files use each their own local React. Since having two copies of React breaks the app, this makes the npm link approach unusable for me.

[immortal-tofu]

Same issue here. npm dedupe doesn't work with npm link. I didn't find a workaround yet.

[Reinmar]

We've got the same problem in CKEditor. We have multiple plugins requiring each other and the core package where all this is glued together. In this main package I would like to be able to npm link some of the plugins and call npm dedupe in order to flatten the dependency tree. In my case this step is necessary, because I need to build this as this is a client-side app (hence, I need to transpile JS, concatenate and minify what's needed, etc.). Having a non-deduped dependency tree means that I need to include the deduping logic inside my builder. Of course this is doable, but being able to use npm for that would be awesome.

On the other hand, I understand that by default npm cannot "hoist" dependencies from symlinked modules to their parent, because they may have multiple parents. This isn't a thing in my case though, and I think that this is a pretty common case where you work on a certain set of your modules. Perhaps an option could be added to npm dedupe/install so it also works with symlinks?

[patrickheeney]

Has anyone figured out an easy solution for this?

@Reinmar From reading several GH issues, it sounds like you are using gulp to handle some of these scenarios? How did you handle a shared common dependency like React for example?

[gaelollivier]

Same problem here. We're building a webapp that we split in multiple modules. We use webpack to bundle our app.
In development we npm-link our modules to be able to run our webapp from its "main" module. Most of our sub-modules share common dependencies such as AngularJS. Because Angular is installed in each of our sub-dependencies, it ends up being loaded multiple times in our final bundle and that's obviously very bad... Some dependencies just break if we load them multiple times.

I thought npm dedupe could solve our problem but as this issue stated, it doesn't support cleaning "npm-linked" trees.

Here is my webpack-related issue:
webpack/webpack#2134

I actually can't believe that nobody faced this problem before... I'm new to webpack and managing frontend dependencies with npm in general but I was sure it was a common practice, and using npm-link to work on multiple modules at the same time seems a pretty obvious pattern.

Any insights on this please ? :)

[FunkMonkey]

I also stumbled across this issue (especially about the problem that npm dedupe ignores symlinks) and thus created a npm package for it: npm-dedupe-symlinks.

It basically removes the symlinks and moves the according node_modules folders from the symlink targets into the directory structure of the package you want to dedupe. It then simply calls npm dedupe, which can work on a directory structure without any symlinks. Afterwards the node_modules folders (if they still exist after deduping) will be moved back to the original symlink targets and the symlinks will be recreated.

It only operates on one level of symlinks (so only symlinks in the node_modules of the package itself or in a scope) at the moment and will probably blow up if you symlinked the same module multiple times. Still beta, so use it with caution (make a backup of your project first)!

[vjpr]

How about creating a node_modules dir above all your dev dirs. Npm's search alg will recurse up looking for node_modules until your root dir. Then to prevent duplicate requires, you just delete the modules from each of your linked dependencies. You could have a script that would orchestrate this.

A common node_modules means that you will not break other modules that are npm linked to the same module, but don't have it as a peer dependency.

For me I am seeing babel-runtime, lodash, core-js, bluebird required about 5x each on server startup causing 10s server startups. I like to have tons of modules npm linked when working so I can quickly make changes.

Or...has anyone tried override the npm search algorithm...providing a module alias?

I'm going to try it now.

[gaelollivier]

Having a script orchestrating this almost boils down to re-implementing a lot of logic that npm should be responsible for in my opinion. That's really an issue that npm should be able to fix I think but it seems kind of hard with its current design. It's in the core principles of npm to have this tree hierarchy, not a flat one.

Supporting both may be hard to do but that's worth thinking about it I think, as npm is being widely used for frontend now.

[vjpr]

@gaelduplessix It's actually very easy. You basically hook each call to require, and use your own algorithm to return the resolved path. Webpack has this feature in resolve.alias.

I have implemented it and it works well. I have to clean it up and then I will publish it.

My broader goal is that all my modules are symlinked because 1. npm install is incredibly slow, and 2. I want to be able to patch/test/publish modules instantly. This modified resolver, coupled with a git pre-push script is al that is neccessary.

[broth-eu]

@vjpr are there any news on your resolve.alias solution?

[PixelsCommander]

@broth-eu The solution is to add this in main project webpack config:

resolve: {
        alias: {
            'react': path.join(__dirname, 'node_modules/react'),
        },

[donaldpipowitch]

@ehsalazar Could this be reopend? It looks like you can't use $ npm dedupe if one of your dependencies is a symlink. (For me with npm@3.10.6 it hangs forever at the line loadAllDepsIntoIdealTree: sill loadAllDepsIntoIdealTree Starting.)

[vjpr]

@broth-eu @donaldpipowitch

I have released a module called live-perf that will dynamically dedupe your Node.js modules.

It also provides a few different ways to benchmark the time it takes to require modules in your project, and shows where modules could be deduped.

Check out the examples.

Basically it hooks into the require resolver, examines the closest package.json files for each require statement, and then tries to locate a package that satisfies the dependency requirement.

At the moment, the module's initialization, and its deduping logic nullifies the benefits from having deduped modules. In the examples, if two modules take 1s each (2s total) to eval, deduping will mean it would only take 1s total, but the deduping logic costs 1s. This cost grows with the number of require statements used.

This is something I am looking into.

The slow parts are traversing up to find the package.json files, parsing the package.json files, using semver.satisfies, and retrieving the root packages.

I think it just needs some caching amongst other things.

[donaldpipowitch]

Basically it hooks into the require resolver,

It probably wouldn't work for me then. I sadly need that in webpack projects.

[vjpr]

@donaldpipowitch

I'll have to look into webpack support.

It would scan your entire tree for symlinks in node_modules, then we could use the same hook from my lib and use require.resolve and then add these to resolve.alias.

[vjpr]

Good news - looks like webpack resolvers are pluggable. Should be easy to port.

http://webpack.github.io/docs/plugins.html#resolvers
webpack/webpack#110

[donaldpipowitch]

It would scan your entire tree for symlinks in node_modules, then we could use the same hook from my lib and use require.resolve and then add these to resolve.alias.

That sounds like a good strategy! Thank you.

[bfil]

@ehsalazar I think this issue does need to be reopened.

It seems like it is currently not possible to link a module locally which needs to access the dependency of the main module that depends on it.

For example if I try to link a gulp plugin to another module that uses the plugin, that causes the symlink to have a nested node_modules directory and its own gulp dependency.

And trying to run npm dedupe with a symlink does cause npm to hang at:

loadAllDepsIntoIdealTree: sill loadAllDepsIntoIdealTree Starting

It's not that I don't enjoy running npm uninstall - npm publish - npm install after every minor change in one of my modules, I do! Oh wait..

[joelday]

Is there an official "right way" to work on multiple, interdependent local modules at the same time that doesn't suffer from the above mentioned side effects?

[clayne11]

This is a serious problem. How are you supposed to develop interdependent packages efficiently?

Is there a possibility that dependencies could be installed into a parent directory in certain situations?

Example

File structure

➜  link-test tree
.
├── package.json
└── packages
    ├── a
    │   └── package.json
    ├── b
    │   └── package.json
    └── c
        └── package.json

Package dependencies

// package a
{
  "name": "a",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "c": "1.0.0",
    "lodash.flow": "^3.5.0",
    "lodash.range": "^3.2.0"
  }
}

// package b
{
  "name": "b",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "c": "1.0.0",
    "lodash.flow": "^3.5.0",
    "lodash.range": "^3.2.0"
  }
}

// package c
{
  "name": "c",
  "version": "1.0.0",
  "description": "",
  "main": "index.js",
  "scripts": {
    "test": "echo \"Error: no test specified\" && exit 1"
  },
  "keywords": [],
  "author": "",
  "license": "ISC",
  "dependencies": {
    "lodash.flow": "^3.5.0",
    "lodash.range": "^3.2.0"
  }
}

Ideal solution

To develop this efficiently, I want to npm link module c into both a and b. If I do that, however, I'm going to end up with this:

➜  link-test tree -l
.
├── package.json
└── packages
    ├── a
    │   ├── node_modules
    │   │   ├── c -> ../../../../../.nvm/versions/node/v4.5.0/lib/node_modules/c
    │   │   │   ├── node_modules
    │   │   │   │   ├── lodash.flow
    │   │   │   │   │   ├── LICENSE
    │   │   │   │   │   ├── README.md
    │   │   │   │   │   ├── index.js
    │   │   │   │   │   └── package.json
    │   │   │   │   └── lodash.range
    │   │   │   │       ├── LICENSE
    │   │   │   │       ├── README.md
    │   │   │   │       ├── index.js
    │   │   │   │       └── package.json
    │   │   │   └── package.json
    │   │   ├── lodash.flow
    │   │   │   ├── LICENSE
    │   │   │   ├── README.md
    │   │   │   ├── index.js
    │   │   │   └── package.json
    │   │   └── lodash.range
    │   │       ├── LICENSE
    │   │       ├── README.md
    │   │       ├── index.js
    │   │       └── package.json
    │   └── package.json
    ├── b
    │   ├── node_modules
    │   │   ├── c -> ../../../../../.nvm/versions/node/v4.5.0/lib/node_modules/c  [recursive, not followed]
    │   │   ├── lodash.flow
    │   │   │   ├── LICENSE
    │   │   │   ├── README.md
    │   │   │   ├── index.js
    │   │   │   └── package.json
    │   │   └── lodash.range
    │   │       ├── LICENSE
    │   │       ├── README.md
    │   │       ├── index.js
    │   │       └── package.json
    │   └── package.json
    └── c
        ├── node_modules
        │   ├── lodash.flow
        │   │   ├── LICENSE
        │   │   ├── README.md
        │   │   ├── index.js
        │   │   └── package.json
        │   └── lodash.range
        │       ├── LICENSE
        │       ├── README.md
        │       ├── index.js
        │       └── package.json
        └── package.json

Clearly that's not what I want. I'm using the same packages, lodash.range and lodash.flow between all 3 packages, however I'm ending up with 3 version of the exact same lodash.range and lodash.flow.

Idea

In this case, if the versions matched up, what if npm installed the dependencies into the root directory, like so:

➜  link-test tree -l
.
├── node_modules
│   ├── lodash.flow
│   │   ├── LICENSE
│   │   ├── README.md
│   │   ├── index.js
│   │   └── package.json
│   └── lodash.range
│       ├── LICENSE
│       ├── README.md
│       ├── index.js
│       └── package.json
├── package.json
└── packages
    ├── a
    │   ├── node_modules
    │   │   └── c -> ../../../../../.nvm/versions/node/v4.5.0/lib/node_modules/c
    │   │       └── package.json
    │   └── package.json
    ├── b
    │   ├── node_modules
    │   │   └── c -> ../../../../../.nvm/versions/node/v4.5.0/lib/node_modules/c  [recursive, not followed]
    │   └── package.json
    └── c
        └── package.json

Now I only have one copy of each package, and I have c symlinked into both a and b so that it can pick up changes instantly. It could potentially even symlink c into that top-level node_modules folder as well.

Perhaps we could add a flag to mark a certain package.json file as the "root" of the project ("root": true?)

Issues

Same package, different version

If we have the same package with different versions requirements for two dependencies it isn't going to be able to resolve them this way because one would overwrite the other.

Conclusion

We need a solution for this. It takes forever when I make a change in one of my shared packages to fully rebuild it using file:../ relationships between the packages - up to 10 minutes for any code change. I'm not sure if what I'm proposing makes sense but I think we need to come together to try to find a solution.

It's only an issue for development, but developer time is very valuable and it's costing us collectively a lot of it trying to deal with this mess.

[Reinmar]

I don't see Lerna mentioned here: https://github.com/lerna/lerna

It's able to create cross-symlinks between multiple packages developed at the same time and has some other tools – e.g. it automates releasing and publishing all those packages. There are quite notable projects using it – e.g. Babel.

However, it misses one thing – the ability to link those packages from the main package, assuming it requires them too. It's not a big deal, because those symlinks are easy to make on your own but perhaps such a feature could be proposed.

[lextiz]

@ehsalazar @isaacs, could you please reopen this issue or clarify whether having nested packages using "npm link" and "npm dedupe" is meant to be supported?

[abhirathore2006]

the solution which we use to remove the duplicate is define everything as external in webpack config and then those actual dependencies will be only bundled with one main module.

[hadirsa]

@abhirathore2006, can you provide a sample configuration? I have spent my days on this problem but the problem is not solved.

[abhirathore2006]

in webpack config define externals

 /** utility function **/
function getExternals(lib) {
  return {
    commonjs: lib,
    commonjs2: lib,
    amd: lib,
    root: lib
  }
}

/** webpack config option **/
externals : {
  "axios": getExternals('axios'),
  "immutable": getExternals('immutable'),
  "lodash": getExternals('lodash'),
  "moment": getExternals('moment'),
  "react": getExternals('react')
}