cmd/go: 'go mod tidy' does not add indirect private dependencies

[thanasik]

What version of Go are you using (go version)?

go version go1.11.1 linux/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

GOARCH="amd64"
GOBIN=""
GOCACHE="/home/karysto/.cache/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="linux"
GOOS="linux"
GOPATH="/home/karysto/go"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/linux_amd64"
GCCGO="gccgo"
CC="gcc"
CXX="g++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fmessage-length=0 -fdebug-prefix-map=/tmp/go-build201986556=/tmp/go-build -gno-record-gcc-switches"

What did you do?

I have a private repo I'm working on called parent_repo, it imports another private repo I wrote called child_repo. child_repo imports another private repo I wrote called grandchild_repo. All of these repos have go.mod files. In this case, grandchild_repo is an indirect dependency of parent_repo.

When I run go mod tidy on parent_repo, child_repo appears, but grandchild_repo does not. After adding grandchild_repo manually to parent_repo's mod file (with an associated replace directive to find it locally), go mod vendor and go build -mod=vendor works perfectly, the build succeeds. The problem is that if I do go mod tidy on parent_repo, my manually added grandchild_repo entry is removed, and my builds fail.

My use case is wanting to automate gathering dependencies to vendor before running a CI pipeline, but I can't right now because of this issue.

What did you expect to see?

I expected to see go mod tidy add my indirect dependency to go.mod (so that I didn't have to add it manually) and the build to succeed as a result.

What did you see instead?

I see go.mod tidy remove the grandchild dependency and the build failing

[andybons]

@bcmills

@gopherbot needsinvestigation modules

[myitcv]

@karysto taking your example; given child_repo is a module, then its dependencies (including grandchild_repo) will not appear in parent_repo. The dependency relationship still exists: go list -m all will show you all your dependencies in case you need to see that list.

This is working as intended.

[thanasik]

@myitcv thanks for taking a look. If the required dependencies (in this case grandchild_repo) aren't put into in go.mod by go mod tidy, then go build fails. Is there a workflow or tool I'm missing that automates adding these sub-dependencies to go.mod to allow the build to succeed?

[thepudds]

@karysto sorry for just a quick comment, but does it seem like #26241 might be related?

Also, would you be able to put together 3 or so short but representative go.mod files that capture the essence of what you are seeing?

(A full blown repository on github that shows what you are seeing would be even better, but maybe to start some sample go.mod files could help make the nature of the problem more concrete).

[myitcv]

@karysto If you're seeing a build failure then there is something else going on, because grandchild should not be required in the parent's go.mod:

$ cd $(mktemp -d)
$ mkdir parent child grandchild
$ pushd grandchild
/tmp/tmp.VJ58bS5pTj/grandchild /tmp/tmp.VJ58bS5pTj
$ go mod init example.com/grandchild
go: creating new go.mod: module example.com/grandchild
$ cat <<EOD >main.go
package grandchild

func Count() int {
        return 5
}
EOD
$ go mod tidy
$ popd
/tmp/tmp.VJ58bS5pTj
$ pushd child
/tmp/tmp.VJ58bS5pTj/child /tmp/tmp.VJ58bS5pTj
$ go mod init example.com/child
go: creating new go.mod: module example.com/child
$ cat <<EOD >main.go
package child

import "example.com/grandchild"

func Children() int {
        return grandchild.Count()
}
EOD
$ go mod edit -require=example.com/grandchild@v0.0.0 -replace=example.com/grandchild=$PWD/../grandchild
$ go mod tidy
$ cat go.mod
module example.com/child

require example.com/grandchild v0.0.0

replace example.com/grandchild => /tmp/tmp.VJ58bS5pTj/child/../grandchild
$ popd
/tmp/tmp.VJ58bS5pTj
$ pushd parent
/tmp/tmp.VJ58bS5pTj/parent /tmp/tmp.VJ58bS5pTj
$ go mod init example.com/parent
go: creating new go.mod: module example.com/parent
$ cat <<EOD >main.go
package main

import (
        "example.com/child"
        "fmt"
)

func main() {
        fmt.Printf("My child has %[1]v children; hence I have %[1]v grand children\n", child.Children())
}
EOD
$ go mod edit -require=example.com/child@v0.0.0 -replace=example.com/grandchild=$PWD/../grandchild -replace=example.com/child=$PWD/../child
$ go mod tidy
$ cat go.mod
module example.com/parent

require example.com/child v0.0.0

replace example.com/grandchild => /tmp/tmp.VJ58bS5pTj/parent/../grandchild

replace example.com/child => /tmp/tmp.VJ58bS5pTj/parent/../child
$ go run .
My child has 5 children; hence I have 5 grand children

Notice I'm using replace directives here (per @thepudds' comment above) because I haven't published the dependencies. Hence I do need a replace for grandchild until I do publish them, but not a require.

[thanasik]

@thepudds thanks for the response, I'll setup a repo demonstrating the issue so that it can be reproduced.

@myitcv I agree, grandchild_repo should not be needed, but for some reason the build fails if it's not included in the parent_repo go.mod. I have the set of replace directives for the local dependencies in the go.mod as well, the issue persists despite them. Do you have an idea of what other possible issue this could be related to that I might be missing?

[myitcv]

@karysto

Do you have an idea of what other possible issue this could be related to that I might be missing?

Not without a repro, no.