Large Number of Diffs After init-ing from gvt

[RussellRollins]

What version of dep are you using (dep version)?

What dep command did you run?

(Sorry, I don't have the exact output recorded here, and can't easily re-init. I can go back in time a bit if necessary)

What did you expect to see?

A new Gopkg.toml that roughly matches my old vendor/manifest file

What did you see instead?

A new Gopkg.toml that changes many dependency revisions, and a large number of diffs in my vendor dir.

After some investigation of the changes, my theory is that dep init only checks against manifest in the case of direct dependencies and not transitive dependencies (which are allowed to float). Is this the case? If this is the case, could the documentation be updated to better explain that? It is harder to get team buy in for the new tool when we're looking a massive diffs in the PR.

I have searched the docs and other issues for similar questions/answers about large diffs, and the best I can get from the docs seems to be:

Pick the highest compatible version for each dependency

Which doesn't quite answer my question/resolve my issue.

If my theory about only transitive dependencies changing is not correct, are there steps I can take to ensure a better match between my current vendor dir and the one dep init creates?

[carolynvs]

Import during init preserves constraints that are: (note the below applies to constraints in Gopkg.toml, not the locks in Gopkg.lock)

• on direct dependencies (I have Proposal: Allow transitive constraints #1489 open to change this behavior for all dep commands)
• not pinned to a specific revision, so branches, tags or semver ranges
• that do not invalidate the locked revision from the imported config. So if the imported config has rev=abc123 and a constraint that would cause dep to immediately discard that locked revision, dep init does not import that constraint.

Init does make every effort possible to preserve the imported revisions in the Gopkg.lock file. So you may see a project not listed in the Gopkg.toml but still listed in Gopkg.lock, and that is by design.

The locked revision for a transitive dependency is not discarded unless something else during solve forced dep to pick a different revision. The goal is to keep as many, if not all, of the locked revisions exactly the same after importing (though the manifest will not have those exact revisions pinned in it, if you want them pinned, add them by hand afterwards).

There is a new feature coming soon that will warn you when dep could not preserve a locked revision during init (#1475) that will help people identify during init when dep wasn't able to do that.

If you can provide the output, I can help figure out why dep did what it did, and if there is a bug, and how to best preserve the original configuration.

[RussellRollins]

Thanks for the response.

We decided to move past the issue, trust our build and test pipeline, and go ahead and make the dep cutover. Things mostly went smoothly, but let me try and provide some output around an example that shows both my original concern, and ultimately caused an issue for us.

Our old gvt manifest

Output from dep init -v

Resulting dep Gopkg.lock

We ran into problems around a transitive dependency of Docker's:

Docker, our direct dependency, was not allowed to change. runc/libcontainer/system, its import, was.

BEFORE dep init

github.com/docker/docker/pkg/archive
"revision": "63a9ea58eb4c704ed7d7dab950f12f708cf220f7"

github.com/opencontainers/runc/libcontainer/system",
"revision": "8a055cad4b24c9987be2f10ab2d98923e5ae9b1b",

AFTER dep init

name = "github.com/docker/docker"
revision = "63a9ea58eb4c704ed7d7dab950f12f708cf220f7"

name = "github.com/opencontainers/runc"
revision = "1a124e9c2da68c867ed2c4f4ff19f0cd95cda0cd"

The version dep changed it to corresponds to: https://github.com/opencontainers/runc/releases/tag/v0.0.8

The version we were using before corresponds to opencontainers/runc@8a055ca, which is tagged with 1.0.0-rc4.

The version that Docker requests at our SHA is: https://github.com/moby/moby/blob/63a9ea58eb4c704ed7d7dab950f12f708cf220f7/vendor.conf#L66

Which corresponds to: https://github.com/opencontainers/runc/blob/1c81e2a794c6e26a4c650142ae8893c47f619764/VERSION#L1 1.0.0-rc4 again.

This example shows how dep changes our /vendor'd files (it changed the version of opencontainers/runc we had by quite a bit), how that might be scary (will our builds still be workable?), and how it actually failed (projects using those files failed to build on MacOS due to changes to "runc/libcontainer/system".RunningInUserNS).

I hope that gives a more concrete example of why these large changes on init were worrying, and a way that it could cause failure.

Thanks again for the help, we're excited to have gotten on to dep, and love all the hard work that's been put into it.

[carolynvs]

Thanks for the extra info! I'll dig into this and let you know what I find.

[mvdan]

Dep was officially deprecated earlier this year, and the proposal to archive this repository was accepted. As such, I'm closing outstanding issues before archiving the repository. For any further comments, please use the proposal thread on the Go issue tracker. Thanks!