Handling dependencies of dependencies and their resolution

[victornoel]

Do you want to request a feature or report a bug?

Bug

What is the current behavior?

This has been made visible via angular/angular-cli#4611 and angular/angular-cli#4787.

Basically, there are some dependencies to:

• "@angular/cli": "1.0.0-beta.32.3" (as a devDependencies)
  • which depends on "@angular/compiler" ">=2.3.1 <5.0.0"
• "@angular/compiler": "4.0.0-beta.8" (as a dependencies)

The problem is that yarn resolves @angular/cli dependency to "@angular/compiler" "2.4.8" which creates problems with angular-cli.

If the current behavior is a bug, please provide the steps to reproduce.

See angular/angular-cli#4787 (comment):

sudo yarn global add @angular/cli
ng set --global packageManager=yarn
ng new MyYarnApp -ng4
cd MyYarnApp
ng serve or ng build

What is the expected behavior?

I would either expect yarn to:

• consider that since "@angular/compiler": "4.0.0-beta.8" is explicitly expressed in the package.json, it should use it also for dependencies dependencies if it matches (which is the case here).
• at least give me a way to force dependencies dependencies versions if I need to.

Please mention your node.js, yarn and operating system version.

yarn 0.20.3, node 6.10.0, latest archlinux

[victornoel]

For the record, I tried to use the resolutions field to force the version of the concerned dependencies, but they are just ignored…

[kylecordes]

It is possible to work around this by running yarn, manually editing yarn.lock, deleting node modules, and rerunning yarn. Not fun.

[coryrylan]

Was able to get this running with what @kylecordes said as well

[moimikey]

i've dealt with this too unfortunately. my solution has been to simply regenerate the lock file :/, even if it means rm'ing it and regenerating it. i install my dependencies flat, so during the "choose version" question and answer game, if I accidentally select the wrongly resolved semver, i'm stuck in that choice and need to rebuild yarn.lock again to get the correct version resolved.

[maxime1992]

@moimikey erasing the yarn.lock is not an option here (for those who don't use flat). Indeed, upgrading to angular v4 while angular-cli still have a dep on angular v2 give something like that in yarn.lock :

"@angular/core@4.0.0-rc.1", "@angular/core@>=2.3.1 <5.0.0 || >=4.0.0-beta <5.0.0":
  version "4.0.0-rc.1"
  resolved "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.1.tgz#7f87b7696b407476e45d6d3c1880a50d5afbb6e3"

"@angular/core@^2.0.0":
  version "2.4.8"
  resolved "https://registry.yarnpkg.com/@angular/core/-/core-2.4.8.tgz#bf1a4fc324827516e6c3222047a9b2cbdaee6976"

So we have to change it by hand to

"@angular/core@4.0.0-rc.1", "@angular/core@>=2.3.1 <5.0.0 || >=4.0.0-beta <5.0.0", "@angular/core@^2.0.0":
  version "4.0.0-rc.1"
  resolved "https://registry.yarnpkg.com/@angular/core/-/core-4.0.0-rc.1.tgz#7f87b7696b407476e45d6d3c1880a50d5afbb6e3"

[mattma]

@maxime1992 @splintercode the issue has been fixed and works as expected

[victornoel]

@mattma no it wasn't, or at least not the problem underlined here in this issue, see my comment on the angular-cli thread.

But after some thought, I'm not sure it make really sense to consider this situation a "bug"... anyway I will leave this decision to yarn dev.
Though I think that the resolve property of package.json should have helped with the situation, which isn't the case, so this can be called a "bug" :)

[mattma]

@victornoel thanks for the updates. I agree with you.

[cansin]

@mattma is there a separate issue open for "resolutions bug" that I could subscribe to? I searched through the list but couldn't find any. Perhaps this is the right one to subscribe to? Thank you!

[cansin]

Shoot. I meant to ask above question to @victornoel ^^

[victornoel]

@cansin sorry, I saw it, but anyway I'm not sure how to answer that… what's your objective here? You just want to know about "resolutions bug" in general?

[cansin]

@victornoel I have a package.json where I try to fixate certain library versions e.g.

  "dependencies": {
    "jquery": "1.11.3",
     ... and some other dependencies
  },
  "resolutions": {
    "jquery": "1.11.3",
  },

But I still end up with a yarn.lock file that has multiple entries for jquery package. I was under the impression adding a resolution for a package would result in:

1. Either raise an error when resolution cannot be met,
2. Or fix the jquery version to 1.11.3 no matter what.

But in my case yarn silently ignores the resolution and decides to resolve a sub jquery dependency with a concrete jquery@2.2.4.

[cansin]

Wait for a second, I was right about asking the question to @mattma in the first place. As they claimed the issue is fixed, and not you @victornoel . Sorry about the confusion.

Anyway @mattma care to comment re: above?

[maxime1992]

@cansin a simple fix is to edit yarn.lock while this issue is still open.

For example, in my project I use Typescript (v2.3.1) but sub dependencies relies on others Typescript versions. No matter what I try to do in package.json, my yarn.lock always ends up into something like that :

...

typescript@2.3.1:
  version "2.3.1"
  resolved "https://registry.yarnpkg.com/typescript/-/typescript-2.3.1.tgz#e3361fb395c6c3f9c69faeeabc9503f8bdecaea1"

"typescript@>=2.0.0 <2.3.0":
  version "2.2.2"
  resolved "https://registry.yarnpkg.com/typescript/-/typescript-2.2.2.tgz#606022508479b55ffa368b58fee963a03dfd7b0c"

...

I just edit it by hand like that :

...

typescript@2.3.1, "typescript@>=2.0.0 <2.3.0":
  version "2.3.1"
  resolved "https://registry.yarnpkg.com/typescript/-/typescript-2.3.1.tgz#e3361fb395c6c3f9c69faeeabc9503f8bdecaea1"

...

So yes, it ain't perfect solution to edit yarn.lock by hand but it's working for now 😉

[cansin]

@maxime1992 thanks for the suggestion. Though since I am working with a relatively large team (>80 eng.), I cannot manually keep yarn.lock synced with the resolution requirements :(

[maxime1992]

Indeed it's far away from a perfect solution but it's the only fix I could come up with right now ...
Maybe only one person should be in charge of upgrading your dependencies and you should keep an eye on yarn.lock in PR if you have some to make sure of this consistency ...

Of course, let's hope it's fixed soon because it's a real problem ;)

[victornoel]

Unfortunately, this workaround does not work anymore with yarn 0.24.x since #2629 was solved with #3106...

@bestander this issue is becoming critical now, because it's really complex to handle manually situations where the resolved dependencies are incorrect (because of faulty dependencies package.json).

My use case is the following:
One of my dependency (angular-cli) insists to use the following version typescript@>=2.0.0 <2.3.0 even though I KNOW that I can use typescript 2.3.x with it (and need because of some of the newest features). Thus, I set up typescript@2.3.2 in my package.json, next to angular-cli.
So I find myself with two typescript versions (and it's a big package) installed and the wrong one used by angular-cli…

[victornoel]

And for the record, it's not always for convenience reasons that this happens, the same happened during the transition between angular v2 to v4, when many dependencies forced the dependency to angular 2 even though v4 was API compatible with v2.

[bestander]

If a dependency depends on typescript@>=2.0.0 <2.3.0 and it works with typescript 2.3.x then ask the library maintainer to loosen the dependency semver or fork it an loosen it yourself.
Why should yarn violate dependency requirements?

[maxime1992]

@bestander we can't ask every maintainer to do that nor do it ourselves.
Why a resolution in package.json would be so problematic ?

[bestander]

Maybe I don't understand it right but looks like that Yarn is resolving the dependencies correctly according to semver.
What is the proposal for Yarn to do?

[victornoel]

@bestander we want the same thing that --flatten does, but on a per-package basis.

It's a feature used a lot in maven for example. Sometimes you use libraries that are not maintained anymore and that depends on buggy versions and the only choice you have is to force a version for one of its dependency.

Something like a resolve field where you can enforce a given version for a given dependency's dependency would be such an answer (and I believe that's what --flatten does, but then you need to do it for one and every dependency of your project instead of a select number).

[victornoel]

(by the way, it's more a feature request than a bug, except if resolve was meant to solve this situation).

[bestander]

Ok, I understand it now.
This is a feature request that Yarn could satisfy if there is enough demand to override semver restrictions.
How many packages are there with incorrect peerDependencies and that are unmaintained?
I am still not sold that no one can just do a fork and publish fixes.

[kylecordes]

How many packages are there with incorrect peerDependencies and that are unmaintained

Anecdotally, many. npm.js contains 350,000 packages. I wonder what percentage of them are maintained and have up to date dependencies?

do a fork and publish fixes.

Anyone who needs to can fork package A, creating package A2 with various fixes and update. Unfortunately, convincing the maintainers of package B, if it is maintained at all, to change their dependency to the new A2 fork, might or might not be possible.

[bestander]

Anecdotally, many. npm.js contains 350,000 packages. I wonder what percentage of them are maintained and have up to date dependencies?

I wonder the same, it is possible to fetch all package metadata from npm couchdb and do a query, so actual number does not need to be a speculation.
This feature is required only for packages that require outdated peer dependencies that are actually compatible with their new versions, I don't think there are as many.

[victornoel]

what is simpler? override the version of a dependency from your project or fork another project, change the dependency, fork all the dependencies of your project depending on the forker dependency and do the same, and so on, publish all of that to npm and then release you project with the forked dependency? :)

This actually raises the question of propagating overridden dependencies to project that depends on your project, but I think that for now it's out of the feature request.

Concerning your comments @bestander, it's not only about unmaintained dependencies, that was just one use case.
As a big user of dependency management software and in particular maven, the feature we are requesting is something really obvious when you are used to build big software based on frameworks and stuffs like that. You start to get sets of dependencies that work together, but sometimes the package maintainer are being for example conservative with dependencies version (such as the angular-cli and typescript case), or you need some specific version for this or that reason, and so on.
npm gives you the possibility to override dependencies version if I'm not mistaken, no? See https://nodejs.org/en/blog/npm/managing-node-js-dependencies-with-shrinkwrap/
That's exactly the kind of thing you can do with maven too (but in a different way).

Actually I'm a bit out of arguments because I'm surprised that the feature is so difficult to be considered useful ^^ If needed I will take the time to find some more :)

[bestander]

Thanks for sharing your point, @victornoel, those are the sad parts of a decentralized dependency ecosystem that we have.

My concern was that this looks like a patch that masks a symptom of a serious issue and every one in the same situation has to apply this patch instead of fixing the cause.

However I am on your side now :)
Would you want to drive an RFC and implementation?
I think we might want a more generic feature that allows overriding a particular semver with another one per package.

This would allow replacing, for example typescript >=2.0.0 <2.3.0 with a ^4.0.0 or jest ^18.0.0 with 18.0.5.

[victornoel]

@bestander so to be sure I understand (before I try to make an RFC, I'm not so sure I will be able to help for implementation though ^^), you propose to be able to express, in the package.json of a project, that typescript >=2.0.0 <2.3.0 should be mapped to typescript ^4.0.0 for example?

Something like that:

"devDependencies": {
    "@angular/cli": "1.0.3",
    "typescript": "2.2.2",
    "more dependencies..."
  },
  "resolutions": {
    "typescript@>=2.0.0 <2.3.0": "typescript@2.2.2"
  },

What I don't like with this is that I have to know that @angular/cli is expressing its dependency to typescript as typescript@>=2.0.0 <2.3.0.

I feel that (but it's only based on my maven experience and current needs, I may be missing some other use cases) it would be nicer to be able to say something like this in the package.json: for a given dependency (e.g., angular-cli), typescript should have the spec ^4.0.0. In this case it wouldn't impact the dependencies of other dependencies.
This is similar to what is done with npm/shrinkwrap if I understood right.

Something like that (note that then it is necessary to differentiate dev and non-dev dependencies I think...):

  "devDependencies": {
    "@angular/cli": "1.0.3",
    "typescript": "2.2.2",
    "more dependencies..."
  },
  "devResolutions": {
    "@angular/cli": {
      "typescript": "2.2.2"
    }
  },

An alternative (very similar to how maven works, and also how --flatten with the resolutions field works if I understood right) is to specify for the whole project which version of a given dependency should be used, for all dependency.

Something like that:

  "devDependencies": {
    "@angular/cli": "1.0.3",
    "typescript": "2.2.2",
    "more dependencies..."
  },
  "resolutions": {
    "typescript": "2.2.2"
  },

The last one is the simplest in terms of expressiveness AND understanding.
The second one is more complex but allows to express more fine-tuned things.
The first one is kind of a mix of the other two, but seems to get the oversimplicity of the last (because you specify something for the whole package and not for each of the package's direct dependencies) and the complexity of the second (because you are actually targeting only some of the dependencies, those matching the specs, but you can't even choose which one). Honestly I don't really like it, sorry ^^

What do you think?

[bestander]

Propose in an RFC the one you think will be the best for most people.

Then others from the community can comment and we'll agree on something.
I think all have reasonable pros/cons.

[victornoel]

ok, thanks!

[cansin]

@victornoel @bestander thanks for trying to figure this out. @victornoel I agree that your last proposal makes the most sense. i.e.

  "devDependencies": {
    "@angular/cli": "1.0.3",
    "typescript": "2.2.2",
    "more dependencies..."
  },
  "resolutions": {
    "typescript": "2.2.2"
  },

resolving to 2.2.2.

[victornoel]

@cansin yep, I think am going for this one, and it goes well with the current flatten behaviour, so it's more of an improvement than a complete new feature, so less chance to confuse people and maybe easier to implement.
@bestander I will make a PR for an RFC next week-end.

[bestander]

Looking forward, when maintaining OSS builds in React Native I remember quite a few situations when a breaking patch in a deep dependency broke React Native for everyone and there was no other way out except for begging the dependency maintainers to release a new patch.

[victornoel]

RFC submitted: yarnpkg/rfcs/pull/68 :)

[e1himself]

Hello,
We have a similar problem. But IMO it's more like a bug than a missing feature.

Setup

So we have a package fork as a top-level dependency.
We have a few packages that have that package as a dependency.

// package.json
{ 
   dependencies: {
      foo-bar: github:acme/foo-bar-fork#0.10.0
      foo-bar-plugin: ^0.15.2    
   }
}

// node_modules/foo-bar-plugin/package.json
{ 
   dependencies: {
      foo-bar: ^0.9.0
   }
}

Expected behavior:

• yarn installs foo-bar package from a fork repository (v0.10.0)
• foo-bar-plugin package dependency is satisfied by a forked package installed on a top level
• there is a single version of foo-bar package in the top level node_modules folder

Actual behavior:

• yarn installs foo-bar package from a fork repository (v0.10.0)
• foo-bar-plugin package dependency is satisfied by stock foo-bar package (v0.10.1)
• we have multiple versions of foo-bar package (fork + stock) on different levels of node_modules

---

So my question: is this a related problem or do I need to submit another issue report?

Thanks.

[victornoel]

@e1himself submit another bug report :)

[BYK]

Closing since the main request seems to be resolved with #4105.