[feature] improve resolution deduping

[bestander]

Do you want to request a feature or report a bug?
feature

Tentative about making it into 1.0 release because I think it is an important improvement over the existing experience.

What is the current behavior?

1. Final deduping

As described in #3775 whether Yarn will download 2 different versions of left-pad depends on the order dependencies are resolved (it's alphabetic BFS).

Instead we should have a post resolution step that would try to dedupe all versions of a single package to the minimum after the dependency tree was resolved.

I started this https://github.com/yarnpkg/yarn/pull/3563/files#diff-80a032b8112ef9aa0ea41ee304a6024aR453 but never have finished.

2. Turn off deduping

At the same time we probably could have the opposite option - force Yarn to use latest dependencies everywhere and not optimize resolution at all.

3. Don't dedupe lockfiles on install

Another aspect - we don't want to dedupe dependencies if they are already in the lockfile because Yarn guarantees same node_modules for the same lockfile.

Please mention your node.js, yarn and operating system version.
Yarn 0.27

[bestander]

Up For Grabs - comment here if you want to give it a try.

[magicmark]

I'm looking at this now, happy if someone beats me to the punch though :)

[SimenB]

1. Final deduping

This sounds ideal, and how I would expect it to work. Might be behind a flag, though, in case some transitive deps has to be in a specific version? If behind a flag, I'd expect a warning printed that I can dedupe if I want to. ("warning pass --dedupe in order to deduplicate left-pad" or something like that).

2. Turn off deduping

Current behavior, right?

3. Don't dedupe lockfiles on install

On install I agree it shouldn't dedupe. But that mandates that yarn dedupe becomes a command. If not, there is no way to dedupe (except manually editing the lockfile).

[bestander]

Turn off deduping

Current behavior, right?

It should still dedupe but less aggressively, could depend on the dependencies order.

I agree with the other statements

[mtraynham]

I know this is a feature request, but did deduping change between 26.1 and 27.5? After the upgrade, it seems my Webpack build is pulling in multiple versions of Angular and jQuery. I'm not entirely sure how to go about actually resolving this issue besides manually editing the lockfile.

The angular example:

angular@1.6.1:
  version "1.6.1"
  resolved "https://registry.yarnpkg.com/angular/-/angular-1.6.1.tgz#a7b6d763a74c157325692619e97257e69d9b6a27"

angular@>=1.2.0, angular@^1.5.8:
  version "1.6.5"
  resolved "https://registry.yarnpkg.com/angular/-/angular-1.6.5.tgz#37f788eebec5ce2e3fa02b17bbcb2a231576a0d6"

The latter here is a transitive dependency pulled in from another library, which allows angular 1.X greater than 1.2. The former is my own version, but this version seems to satisfy the range of the dependency as well.

The jQuery dependency is similar. Former here is my own supplied version, latter being transitive:

jquery@3.1.1:
  version "3.1.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.1.1.tgz#347c1c21c7e004115e0a4da32cece041fad3c8a3"

jquery@>=1.10, jquery@>=1.6, jquery@>=1.8.0, jquery@>=1.9.1:
  version "3.2.1"
  resolved "https://registry.yarnpkg.com/jquery/-/jquery-3.2.1.tgz#5c4d9de652af6cd0a770154a631bba12b015c787"

On the topic of the feature request, can we just reuse resolutions to handle these discrepancies, flattening only a subset of packages? I expect in a standard NodeJS execution environment, it would be unnecessary, but when packaging for the web, it's usually good to only have a single version of a dependency (e.g. only flatten some dependencies if necessary).

[SimenB]

@mtraynham I keep around an older installation of yarn in order to dedupe, as it's broken in newer versions of yarn.

You can manually edit the lockfile to dedupe it as well.

[markwoon]

I'm no longer seeing the dedupe problem with 0.27.5.

[xdumaine]

I'm on 0.27.5 and still have duplication problems. I have a dependency tree like this

app
|-- shared-dep@^1.0.0
|-- dep-1
|   |-- shared-dep@^1.1.0
|-- dep-2
     |-- shared-dep@^1.1.0

And when I yarn upgrade shared-dep (say when 1.2.0 is released) it causes duplication in yarn.lock where the resolution for shared-dep doesn't change for the version pulled in by dep-1 and dep-2 and it adds a new (duplicate) resolution for shared-dep so that now it's pulling in 1.1.0 and 1.2.0.

[magicmark]

There is a currently failing test (skipped) in master that should serve as the source of truth for if this is still happening.

#3779

[OliverJAsh]

I can confirm the behaviour described by @xdumaine still occurs on Yarn v1.0.2.

The only workarounds I know of are:

• Manually dedupe by updating lockfile by hand
• Uninstall and then reinstall all dependencies using the shared dependency

[BYK]

@OliverJAsh #4488 makes that test in #3779 passing. Maybe you'd like to try the build for that PR: https://5262-49970642-gh.circle-artifacts.com/0/home/ubuntu/yarn/artifacts/yarn-legacy-1.0.2.js

[Bnaya]

@BYK i checked it on my project and i still get duplicated deps
This is what i get after yarn upgrade with the version you gave:

-"@types/react@*", "@types/react@^15.6.0":
+"@types/react@*":
   version "15.6.0"
   resolved "https://registry.yarnpkg.com/@types/react/-/react-15.6.0.tgz#3b3e92a60b65045e4df48fda324de84d83addbc4"
 
+"@types/react@^15.6.0":
+  version "15.6.2"
+  resolved "https://registry.yarnpkg.com/@types/react/-/react-15.6.2.tgz#2c8495aa853cb37591d0046e9afe544fb837c612"
+

[mxmul]

@Bnaya #4488 intentionally does not rewrite the lockfile, because of the concerns raised in #79 – but I would expect it to produce a more flat lockfile when generating one from scratch (rm -rf yarn.lock && yarn). I think yarn may also need a new command (yarn dedupe?) to run the same optimization across the entire lockfile.

[scinos]

@OliverJAsh @Bnaya You might want to try https://www.npmjs.com/package/yarn-tools to automatically deduplicate the lockfile.

[Bnaya]

Will give a look, thanks!

[rarkins]

Why is a third party tool needed for de-duplication anyway? Ie in which circumstances does yarn generate "correct" lock files that require de-duplicating? This bit has me confused.

[Bnaya]

@rarkins, for example:

yarn init -y
yarn add @types/react@~15
 yarn add @types/react-virtualized

Gives you

# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


"@types/react-virtualized@^9.7.4":
  version "9.7.4"
  resolved "https://registry.yarnpkg.com/@types/react-virtualized/-/react-virtualized-9.7.4.tgz#c8f8ab729abca03fa76deb0eef820a5daee22129"
  dependencies:
    "@types/react" "*"

"@types/react@*":
  version "16.0.7"
  resolved "https://registry.yarnpkg.com/@types/react/-/react-16.0.7.tgz#f85b6c33c988a1631e2f32fedae71ec6d9718a0d"

"@types/react@~15":
  version "15.6.4"
  resolved "https://registry.yarnpkg.com/@types/react/-/react-15.6.4.tgz#3bb57bd43183a05919ceb025a264287348f47e9d"

And usually after yarn upgrade you can get this dups if you are upgrading a direct dependency that is also dependency of dependency

[mxmul]

@rarkins: Here is a pretty minimal repro:

$ mkdir a b
$ cd a && yarn init --yes && yarn add "lodash@>=1.0.0"
$ cd ../b && yarn init --yes && yarn add a@file:../a && yarn add lodash@^3.0.0
$ cat yarn.lock
# THIS IS AN AUTOGENERATED FILE. DO NOT EDIT THIS FILE DIRECTLY.
# yarn lockfile v1


"a@file:../a":
  version "1.0.0"
  dependencies:
    lodash ">=1.0.0"

lodash@>=1.0.0:
  version "4.17.4"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-4.17.4.tgz#78203a4d1c328ae1d86dca6460e369b57f4055ae"

lodash@^3.0.0:
  version "3.10.1"
  resolved "https://registry.yarnpkg.com/lodash/-/lodash-3.10.1.tgz#5bf45e8e49ba4189e17d482789dfd15bd140b7b6"

Note that ^3.0.0 and >=1.0.0 are compatible ranges, but because of the installation order you end up with two versions on disk.

[Bnaya]

I want to point out that yarn-tools from
#3778 (comment)
Works very well

[StephanBijzitter]

It's been a while since this issue has had any activity, but this is still a problem with the current (1.3.2) release, as seen here: dependabot/dependabot-core#243

[Bnaya]

My current workflow with yarn upgrade is:

yarn upgrade-interactive ...
yarn install # this is, in a lot of cases changes the yarn.lock file after yarn upgrade.
yarn-tools fix-duplicates yarn.lock > yarn.lock.fixed && mv yarn.lock.fixed yarn.lock;
yarn install  # this is actually adding more deduping  after  yarn-tools fix-duplicates

To see the changes on my lock file after each change i'm git adding or even committing them.

[klis87]

@Bnaya Interestingly, my case mentioned in #5265 is not picked even by yarn-tools list-duplicates :o