cmd/go: add go mod download to download modules

[sgnn7]

What version of Go are you using (go version)?

go1.11beta2

Does this issue reproduce with the latest release?

Yes (1.11beta2)

What operating system and processor architecture are you using (go env)?

amd64 - linux

What did you do?

Copying just go.mod into an empty directory should allow go mod -vendor (or maybe even go mod -sync) to download the dependencies regardless of having any other files within the directory. Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

$ docker run -it --rm golang:1.11beta2-alpine /bin/sh
/go # apk add -u git
/go # mkdir /foo
/go # cd /foo

/foo # echo "module github.com/sgnn7/test
> 
> require (
>         cloud.google.com/go v0.25.0 // indirect
> )
> " > go.mod

/foo # cat go.mod
module github.com/sgnn7/test

require (
        cloud.google.com/go v0.25.0 // indirect
)

/foo # go mod -vendor
go: finding cloud.google.com/go v0.25.0
go: no dependencies to vendor

/foo # go mod -sync
/foo # # Nothing

What did you expect to see?

go mod should have downloaded go.mod dependencies based on go.mod content regardless if there's code in the repo or not.

What did you see instead?

Neither go mod -vendor nor go mod -sync downloaded anything.

[bcmills]

Without this, you cannot use something like Docker to cache the dependencies independent of the codebase itself.

Can you include some more detail about your use-case? (Why is that a thing you want to do?)

[sgnn7]

@bcmills Sure.

Let's say that you have your code build a Docker image on every git change (most pipelines do this and is probably the biggest use case).

• With dep you could create a Docker layer of cached dependencies by just copying Gopkg.* and fetching the dependencies after which you then copy the full repo and build the project (see here for the original issue for dep on this). This meant that unless you changed the Gopkg files, no dependency re-download was ever needed, making builds with non-dependency changes quick and efficient.
• With current go mod, you cannot just copy go.mod and go.sum and then run go mod -vendor to get this feature parity since it just doesn't download anything without Go files in the repo, forcing the Docker layer to be cache-busted on the smallest of changes in the repo (e.g. README), causing re-download of all dependencies every time which is both slow and extremely inefficient.

[rsc]

Vendor is tailored for the dependencies of a specific module,
not warming a cache.

Go modules have an actual cache, separate from the vendor directory.
If you want to warm that cache in a base docker image, that would be:

go list -e $(go list -m all)

That will trigger a download of every module from go.mod
(and their dependencies) into the GOPATH/src/mod cache,
which should make those downloads available to higher
docker layers, all without using the vendor directory.

[sgnn7]

@rsc Hmm.. I'll give that a try. Is there a reason why this isn't a base go mod command?

[sgnn7]

@rsc: Your cache warming suggestion doesn't work out of the box. The fixed version is below:

go list $(go list -m all 2>/dev/null)

Edit: Nope that doesn't work either fully. This should be working though (may need escaping for Dockerfile):

 go list -e $(go list -m all 2>/dev/null | awk '{print $1}')

^ this is really not something that should be found in Dockerfiles

Edit 2 Third iteration that is the least messy so far:

go list -e $(go list -f '{{.Path}}' -m all 2>/dev/null)

IMO, If this seems like something that should be used often, it probably needs to get into the main CLI. As a sidenote to this hack, go list has no distinction in docs as to why it is the only one that can download modules vs -sync and -vendor.

[rsc]

All the commands downloads modules as needed.
sync and vendor are defined to chase down only the package-granularity dependencies of the packages in the main module.
If there are no packages in the main module, they chase down nothing.
List on the other hand can be asked directly about dependency packages,
and answering requires downloading them to look at the code.
It's certainly not a direct answer, but it should work.

That said, it seems to me that go get -d -m asks for exactly this
operation and fails to deliver it (the -d is ignored with -m).
We should fix that.

[moretea]

I got pretty close with #26610 (comment). To make this work for now, I am using the following Dockerfile:

FROM golang as builder
RUN go get -u golang.org/x/vgo
WORKDIR /go/src/github.com/$ORG/$REPO

# Populate the module cache based on the go.{mod,sum} files.
COPY go.mod .
COPY go.sum .
RUN vgo list -e $(vgo list -f '{{.Path}}' -m all)

# Build all binaries, with -getmode=local, this will _not_ fetch packages over the network.
COPY . .
RUN vgo install -getmode=local ./...

# Runtime image
FROM alpine AS base
COPY --from=builder /go/bin/$PROG /bin/$PROG
ENTRYPOINT ["/bin/$PROG"]

[shanna]

@moretea This takes care of caching downloads but if I add -v to the go install/build the dependencies are being built each pass still so build/install times can still be very slow if you have a lot of dependencies (firebase, grpc, kubernetes etc).

[sgnn7]

Small update for those that are using the go list to fetch the packages: #26631 PR by @marwan-at-work should remove the need to pipe stderr to /dev/null once it gets into the releases.

[marwan-at-work]

@sgnn7 the CL was closed by @myitcv, not sure if it needs to be re-opened. But my own use case for having that CL is resolved by just using stdout.
Thanks

[sgnn7]

@marwan-at-work I don't know why that issue would be abandoned 😢. Maybe it's because the v1.11 GA release date is tomorrow?

As for using stdout, that's what the interim fix uses but it would be nice not to need 2>/dev/null in it to separate stdout from stderr.

Edit: Your PR looked fine to be tbh - I don't really understand the current stdout/stderr separation if stderr shows informational messages.

[myitcv]

@marwan-at-work I closed the CL based on our discussion, yes. Did I prematurely close it? Was there something I missed?

@sgnn7 as things stand, stdout is defined to contain the JSON output; stderr informational messages, with the exit code ultimately used to then determine success or otherwise (with the caveat that -e can alter the handling of erroneous packages, per go help list) of the list operation.

Does that help to clarify things?

[marwan-at-work]

@myitcv CL being closed is good on my side :) Just wanted to ping you about @sgnn7's use case.
Thanks

[sgnn7]

@myitcv Makes sense though I wouldn't say that it increases simplicity/usability cases (compared to most other *nix tooling). Either way, this issue is not for that discussion and thank you for the clarification. 👍

[rsc]

I started looking at go get -d -m but it's a weird combination to redefine.
Instead I am going to add a 'go mod download' that does what you want.
So you could run

go mod download

to warm a Docker image cache.

[marwan-at-work]

@rsc will this new subcommand take care of this issue? #26577
Most notably, go mod download would not download dependencies except just the targeted import path. Or at least has the option to not download dependencies.

Would love to help in any way.

Thanks!

Edit: it does :)

[gopherbot]

Change https://golang.org/cl/128355 mentions this issue: cmd/go: add go mod download

[mattwilliamson]

There is a caveat here, you must have a main package in the directory with go.mod for go mod download.

[marwan-at-work]

@zangbuild no longer the case with go1.12, there's already a beta out if you want to give it a shot.

[mattwilliamson]

Nice. I will try when I can.

[mbana]

Does go mod download also build (and install) the packages?

Edit: Will this do the what I want? go list -m -f '{{.Path}}' all | xargs -I{} sh -c 'GO111MODULE=on go get -x -a {}'

[marwan-at-work]

@mbana go mod download just downloads content for a module, it does not build and install the package nor does it fetch its dependencies.

[mcluseau]

Hi, related question: is there a go mod download-from-vendor for those already having everything in their vendor directory? Thanks!

[bcmills]

@mcluseau, the vendor directory stores individual packages, not complete modules. It is not a suitable input to the module cache.

[mcluseau]

So just to follow up, the best way I've found considering that is to use a Go proxy (athens) and the following Dockerfile:

from golang:1.12.1-alpine3.9 as build
run apk add --update git

env CGO_ENABLED=0
arg GOPROXY

workdir /src
add go.mod go.sum ./
run go mod download

add . ./
run go test ./...
run go install . ./cmd/...

from alpine:3.9
#entrypoint ...
copy --from=build /go/bin/ /bin/

[c0b]

is go mod download designed to pre-download all go.mod defined dependencies? I found a case might be a bug of it, because I have a similar case want to pre-download as cache and build go code later,

$ go mod download
....      [indeed download majority, maybe 99% of dependencies]

$ go build ...
.....       [ but it still leaked a few to go build ... ]
go: finding github.com/alecthomas/template latest
go: downloading github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc
go: finding github.com/alecthomas/units latest
go: downloading github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: extracting github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf
go: extracting github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc

I'm not sure does this mean go mod download failed to parse a complete list of dependencies? leaked a few to go build command, should be filed as a go mod bug? here is using FROM golang:1.12-alpine AS builder

[bcmills]

@c0b, if all dependencies are declared (that is, if go mod tidy is a no-op), then go mod download is intended to download all needed dependencies.

If you find that is not the case, please open a new issue with steps to reproduce (ideally using go1.13beta1 or newer).

[ORESoftware]

So yeah go mod download needs a verbose option, please
#38756

go mod download --verbose

or

go mod -v download

whatever, o/w I am just watching the command hang there forever, hoping that nothing is wrong. some feedback would be nice, like with go install -v