cmd/go: How can I prevent go CLI to check all dependencies before doing actions

[sneko]

What version of Go are you using (go version)?

go version go1.11beta2 windows/amd64

Does this issue reproduce with the latest release?

Yes

What did you do?

I'm trying to execute basic Go commands in a CI/CD environment. The idea is that I have 2 stages representing "BUILD stage" and "TEST stage", so they are totally running in different docker instances while the project directory (= workspace) is the same between both.

In the BUILD stage I first download all project dependencies written in my go.mod and then I run a "go mod -vendor" to copy them from the $GOPATH to a "vendor" subdirectory of my project. Like that in the next TEST stage there is potentially no need to download again all the dependencies.

To tell you more about the context I run some basic commands like:

• Installing global dependecy:
  go get -u github.com/jteeuwen/go-bindata/...

• Make unit tests:
  go test ./...

• Make a coverage test:
  go test -race -cover ./...

What did you expect to see?

I would require these commands to just do their job without trying to check dependencies.

What did you see instead?

Instead, for each of these commands I get something like:

finding PACKAGE_XXXXXX
finding PACKAGE_XXXXXX2
finding PACKAGE_XXXXXX3
finding PACKAGE_XXXXXX3
...

It means that the CLI makes a request to each repository to check the version... and it takes a lot of time while they are already in the vendor directory matching the go.mod requirements!

I guess since there is an empty directory at $GOPATH because this one is not passed from the BUILD stage to TEST stage, the CLI changes its behavior. But there is no reason to...

I succeeded to work around while installing global dependency (go get github.com/jteeuwen/go-bindata/...):

• First I need to specify -u -f parameters according the docs to avoid these checks
• But it triggers an error since my commands are ran inside a Go module. So my workaround is to run instead this:
  GO111MODULE=off go get -u -f github.com/jteeuwen/go-bindata/...

That's not a cleanest way but it works...

At this time I didn't find a solution for either go test ./... and go test -race -cover ./....

One of the solution could be to mount a volume on $GOPATH to have downloaded dependencies from BUILD stage inside the TEST stage but that's not a clean way too. And if you think it is, what is the purpose of go mod -vendor if the CLI cannot manage correctly vendored dependencies?

I hope I was clear in my explanations, don't hesitate to ask more information. Maybe @rsc you have an idea about this since I'm using the latest features coming from vgo.

Thank you 😃 !

[oiooj]

@sneko You can try go test -getmode=vendor ./...

[thepudds]

Hi @sneko, slightly expanding on the answer from @oiooj , here are related some snippets from the module docs on tip:

To allow interoperation with older versions of Go, or to ensure that all files used for a build are stored together in a single file tree, 'go mod -vendor' creates a directory named vendor in the root directory of the main module and stores there all the packages from dependency modules that are needed to support builds and tests of packages in the main module.

To build using the main module's top-level vendor directory to satisfy dependencies (disabling use of the usual network sources and local caches), use 'go build -getmode=vendor'. Note that only the main module's top-level vendor directory is used; vendor directories in other locations are still ignored.

And the doc section on build flags includes listing the commands you asked about (test, build and get) as supporting -getmode:

The build flags are shared by the build, clean, get, install, list, run, and test commands:

...

-getmode mode
module download mode to use. See 'go help modules' for more.

In addition, #26585 and change https://golang.org/cl/126656 describe a related possible enhancement that is targeted at multiple use cases, including specifying -getmode in CI via an environmental variable. Snippet from the CL:

People sometimes want to turn on a particular go command flag by default.
In Go 1.11 we have at least two different cases where users may need this.

1. Linking can be noticeably slower on underpowered systems
   due to DWARF, and users may want to set -ldflags=-w by default.

2. For modules, some users or CI systems will want vendoring always,
   so they want -getmode=vendor (soon to be -mod=vendor) by default.

This CL generalizes the problem to “set default flags for the go command.”

$GOFLAGS can be a space-separated list of flag settings, but each
space-separated entry in the list must be a standalone flag

Is your concern addressed at this point? If so, does it make sense to close this issue?

[thepudds]

And just to be a bit more explicit, one more comment is that as of right now #26585 is still open and change https://golang.org/cl/126656 with $GOFLAGS is still in progress / not yet merged, so of course some chance it might not land and/or change in some way before it lands.

[sneko]

@oiooj @thepudds thank you very much for the help and these explanations! I missed this part in the documentation :s

I've still an issue with the go test -getmode=vendor -race -cover ./... since it tries to find imports in the vendor directory, even if it's an import for a package inside the module...

To be more clear, my project (=module) imports X, Y, Z as external packages, and after running go mod -vendor all of them will be copied in the "vendor/" directory. But my project also includes internal packages (A, B, C), for example, they are imported from the "main" package of the project module. A, B, C won't be copied in the vendor directory but running the coverage test command will look for them in the same directory than external dependencies.

Do I missed something to do...?

EDIT:
Here is the error for a coverage test from the "XXXXXXXXXXXXXXXX/service-user" module :

+ go test -getmode=vendor -race -cover ./...

main.go:6:2: cannot find package "XXXXXXXXXXXXXXXX/service-user/cmd/server" in any of:
	/usr/local/go/src/XXXXXXXXXXXXXXXX/service-user/cmd/server (from $GOROOT)
	/go/src/XXXXXXXXXXXXXXXX/service-user/cmd/server (from $GOPATH)
main.go:7:2: cannot find package "XXXXXXXXXXXXXXXX/service-user/cmd/server/cli" in any of:
	/usr/local/go/src/XXXXXXXXXXXXXXXX/service-user/cmd/server/cli (from $GOROOT)
	/go/src/XXXXXXXXXXXXXXXX/service-user/cmd/server/cli (from $GOPATH)
script returned exit code 1

It's like if it doesn't understand being inside the module that it is looking for... That's the same when running go test -getmode=vendor ./... 😭

[sneko]

My bad...! Since I was putting my final executable in an alpine image I had to build my project with the golang:1.11beta2-alpine to make it working because of well-known musl/glibc issue.

But the flip side is that some unit and coverage tests are not working at all with musl libs so I had to run them inside a "non-alpine" image. That's what I did by putting golang:latest but I forgot that because of the usage of vendoring and modules I had to use golang:1.11beta2.

I'm sorry for posting this stupid "issue" 😀

Thank you again for your help on the main issue!