cmd/go: "get -u" stumbles over repos imported via non-canonical paths #30831
Comments
|
My theory (sorry, it's late, so not sure if it's obviously right or wrong... also, this bug report is poorly written but I'm pretty frustrated and confused by all of this) Because we are allowed circular dependencies on modules, we have two libraries: A@0.1 requires github.com/golang/lint B@0.1 requires A@0.1 A fixes its import to golang.org/x/lint, tags 0.2 B@0.2 updates its A dependency, and now requires A@0.2. But A@0.2 still requires B@0.1 which requires A@0.1 which requires B@v0.0.0-... which requires github.com/golang/lint. Of course, due to MVS, we only choose one version of the dependency at build time. However, So the only to break this cycle is to simultaneously tag new versions of A and B, and have them require each other. (incompatible with CI) In reality, we have more than just two things depending on each other, there's far more. golang.org/x/oauth2, google.golang.org/api/grpc, cloud.google.com/go, google.golang.org/api, go.opencensus.io basically all have dependencies on each other. |
|
/cc @bcmills We've discussed this general issue today (or yesterday) and Bryan said he might be aware of a fix that can be applied to |
|
Something along the lines of this, maybe? (see patch below) The idea is to not try to upgrade something when we already use a newer version. Theory: upgrading from 0.2 to 0.3 is less work than 0.1 to 0.3, and there's no need to check the same package if you already checked a newer version. edit: ah, yes, theory and tests fail because vN+1 may have downgraded one of its deps. @@ -85,6 +89,17 @@ func buildList(target module.Version, reqs Reqs, upgrade func(module.Version) mo
)
work.Do(10, func(item interface{}) {
m := item.(module.Version)
+
+ var skip bool
+ mu.Lock()
+ if v, ok := min[m.Path]; ok && v != "" && reqs.Max(v, m.Version) == v {
+ skip = true
+ }
+ mu.Unlock()
+ if skip {
+ return
+ }
+
required, err := reqs.Required(m)
mu.Lock()Simplified problem... M has a requirement on A@0.1 and B@0.1 "go get -u" still tries to upgrade C, because of M's dependency on B@0.1, even though M won't use B@0.1 or C@0.1 in the build at all, only A@0.1 and B@0.2. Another ecosystem problem is "indirect" entries in go.mod. As evidenced by zipkin including an indirect entry in github.com/golang/lint: A@0.1 requires C@0.0 M requires A@0.2 and B@0.1, indirectly depending on C@0.0 even though it isn't used to build, because MVS upgrades B's A@0.1 requirement to A@0.2. In this case, A = grpc or genproto, B = zipkin, C = github.com/golang/lint |
broady
changed the title
bcmills
added
the
NeedsFix
|
FWIW, I also suspect (a) something is wrong here, or (b) that the behavior here can be improved. Potentially related is #30455 (comment), which asks if
Also potentially related:
|
rsc
changed the title
|
Leaving the "too much upgrading" problem for #26902. It seems like we need a clear signal from the upgraded module that "I used to be this old name". And then if you are resolving the old name and land in the new module claiming "I used to be this old name", the build would automatically interpret the old name as the new name. Maybe that's a per-module signal; maybe it's a per-package signal. It needs to be an explicit signal, though. It's not enough to just "find code at X that seems to be module Y so pretend X means Y" because that breaks people who github fork X into their own account as Y. In that case we don't want go get of the fork to silently redirect back to the original. |
This requires the upgraded module to know its old name. In this specific case, it might be possible, because the canonical import path of lint used to be However, it didn't start enforcing import paths until much later (via import comment for GOPATH mode, and via go.mod file for module mode), so some projects may have started importing it via any other alias, not just Should it be responsibility of an upgraded module to know what were all the non-canonical import paths that some projects in the wild may have imported it as? |
In general that set is well-defined: we must have arrived at the repository somehow (via redirects), so the only possible names for it are things that redirect to it. I would guess that it's pretty rare for folks to set up redirectors for arbitrary repositories just to use them via different import paths. |
|
One short description of the build list (from the cmd/go documentation) is:
It is unlikely that I will describe this exactly correctly, but from the outside it seems that if the inconsistency between Or maybe that is not the right way to think about it -- maybe instead the "bad" consumer version gets added to the build list but is marked that it is bad, but that is not a fatal problem for the overall build as long as the bad version is not the version that gets selected in the end. It seems some type of approach along those line would have solved the original Edit: part of the problem with the current behavior is the degree to which old problems that were fixed a while ago are still a "hangover" for right now. E.g., |
I'm not sure this solves the problem. Suppose there's some requirement on some version of a module that points to the bad pseudoversion of lint that doesn't include the old name. cmd/go still tries to validate that bad version and fails. I think it needs to be treated as a non-fatal error (non-fatal because it doesn't end up in the final build graph).
|
The problem is “does include the new name”, not “doesn't include the old name”. The problem today is that folks depend on revisions of If the |
Understood. I was illustrating a situation* where proposed fix of "module can specify acceptable old names and/or aliases" might not work. I suppose that's an edge case, since it would mean an invalid go.mod file in the first place (e.g., if someone had manually changed golang.org/x/lint to github.com/golang/lint). '* the case where something depends on a version of github.com/golang/lint that doesn't specify its aliases. |
I don't know if
Perhaps a completely different approach would be -- rather than saying the old spelling of lint cannot be upgraded, could the old spelling of lint be allowed to upgrade to the version of lint just before the |
Absent explicit release tags, we would have to scan every commit to the repository to find such a version. |
|
What is the solution? |
As far as I am following, I think this issue here is more about a medium or long term fix by changing the go tool, but that is not happening immediately. However, there is a healthy chance there is a more immediate solution to whatever you are seeing. It probably would make sense for you to review the symptoms at googleapis/google-cloud-go#1359 and golang/lint#436 and see if your symptoms match one of those reports, and either add comments there or perhaps open a new issue. |
Now that tools' dependencies have been cleaned up in golang.org/cl/160837, lint doesn't have any transitive dependencies on modules that import it using the wrong path. It is now safe to reintroduce a go.mod file to this repo. I've checked using dmitshur's instructions in #436 (comment) that lint only appears with its canonical module path in the build list. Updates #436 Change-Id: I6343aa103408b20562e17ea019602b159b899fc6 Reviewed-on: https://go-review.googlesource.com/c/lint/+/166278 Run-TryBot: Michael Matloob <matloob@golang.org> Reviewed-by: Dmitri Shuralyov <dmitshur@golang.org>
|
This has surfaced again in googleapis/google-api-go-client#347 (git.apache.org/thrift.git became github.com/apache/thrift). AFAICT, without the Go tool changing in some way, library owners have to resort to alien things like making libraries rely on non-existent versions as kindly spelled out by @dmitshur here or temporarily omitting dependencies. Both of these solutions require a lot of coordination and releasing multiple tags across multiple. This is pretty painful. :| Are there any other medium-term solutions anyone else knows of? |
golint changed its import path [1], and that along with the advent of modules caused fallout [2, 3] that broke the `go get -u` installation in our makefile/CI build. The tools.go idiom is the currently favored approach for versioning development tools with the module system [4, 5], in a way that `go mod tidy` won't churn them from `go.mod` and the `+build` constraint keeps them out of actual build products. The tools still need to be `go install`ed, within a module `go get -u` is not the thing to do anymore because it upgrades transitive deps of a tool which may change the module's build. It takes like hours of reading discussions to triangulate on these moving targets... [5, 6, 7, 8] jfc how much of life have I spent following the fashion evolution of Go dependency management [1]: golang/lint@c363707 [2]: golang/go#30455 [3]: golang/go#30831 [4]: https://github.com/golang/go/wiki/Modules#how-can-i-track-tool-dependencies-for-a-module [5]: golang/go#25922 [6]: golang/go#27653 [7]: golang/go#27643 [8]: golang/go#30515
This was a bit fun since there was a dependency which hit golang/go#30831 Ended up deleting the go.mod file and then regenerating it to get go get -u to work.
bcmills
changed the title
One of the Ankh dependencies is importing `github.com/golang/lint` which now specifies it must be imported as `golang.org/x/lint`. ```sh go mod edit -replace github.com/golang/lint@latest=golang.org/x/lint@latest ``` golang/go#30831 golang/lint#436
|
Change https://golang.org/cl/220080 mentions this issue: |
Updates golang/go#36460 Updates golang/go#27900 Updates golang/go#26955 Updates golang/go#30831 Updates golang/go#32058 Updates golang/go#32380 Updates golang/go#32419 Updates golang/go#33370 Updates golang/go#33669 Updates golang/go#36369 Change-Id: I1d4644e3e8b4e688c2fc5a569312495e5072b7d7 Reviewed-on: https://go-review.googlesource.com/c/proposal/+/220080 Reviewed-by: Russ Cox <rsc@golang.org>
ghost
mentioned this issue
|
We've given this issue a lot of thought over the last several years. The current best-practice solution for changing a module's import path is to choose at least one of:
Optionally, also leave behind packages in a repo at the old path containing declarations that merely alias (or forward function calls) to the packages at the new path. |
What version of Go are you using (
go version)?Does this issue reproduce with the latest release?
Yes
What operating system and processor architecture are you using (
go env)?go envOutputWhat did you do?
While investigating googleapis/google-cloud-go#1359, golang/lint#436, and opening census-instrumentation/opencensus-go#1064, I've found it impossible to figure out how to get almost any of these packages to build and successfully run
go get -u.It seems to be a never-ending loop of things depending on slightly older things, which ultimately end up depending on the invalid github.com/golang/lint.
For a lot, but not all (see below) of these packages, I was able to track the chain from github.com/golang/lint up to its depender using
go mod graph.Of course, because of MVS, the old version of grpc (and therefore github.com/golang/lint) is never used in the actual build, but for some reason still blocks a successful run of
go get -u.What did you expect to see?
go get -usucceeds, no mention of github.com/golang/lint because it shouldn't be included due to MVS rules.What did you see instead?
The text was updated successfully, but these errors were encountered: