cmd/go: no way to go get source code for editing in module mode

[mholt]

What version of Go are you using (go version)?

$ go version
go version go1.12.4 darwin/amd64

Does this issue reproduce with the latest release?

Yes

What operating system and processor architecture are you using (go env)?

go env Output

$ go env
GOARCH="amd64"
GOBIN=""
GOCACHE="/Users/matt/Library/Caches/go-build"
GOEXE=""
GOFLAGS=""
GOHOSTARCH="amd64"
GOHOSTOS="darwin"
GOOS="darwin"
GOPATH="/Users/matt/Dev"
GOPROXY=""
GORACE=""
GOROOT="/usr/local/go"
GOTMPDIR=""
GOTOOLDIR="/usr/local/go/pkg/tool/darwin_amd64"
GCCGO="gccgo"
CC="clang"
CXX="clang++"
CGO_ENABLED="1"
GOMOD=""
CGO_CFLAGS="-g -O2"
CGO_CPPFLAGS=""
CGO_CXXFLAGS="-g -O2"
CGO_FFLAGS="-g -O2"
CGO_LDFLAGS="-g -O2"
PKG_CONFIG="pkg-config"
GOGCCFLAGS="-fPIC -m64 -pthread -fno-caret-diagnostics -Qunused-arguments -fmessage-length=0 -fdebug-prefix-map=/var/folders/sf/7w4p15_15jb59d7kwy0wglh40000gn/T/go-build796937585=/tmp/go-build -gno-record-gcc-switches -fno-common"

What are you trying to do?

(Added this question, hope that's OK.)

go get a Go project's source code and install its command (the repo contains both a command and library packages) while respecting module dependencies.

In the past we used vendor to pin dependencies, but we'd rather use Go modules. go get has always worked great, even with an empty GOPATH. If possible, we'd like for the project to stay go-gettable.

What did you do?

1. Make an empty GOPATH. (or, as I did, export GOPATH= and ensure no ~/go folder exists)
2. $ GO111MODULE=on go get github.com/mholt/testrepo/caddy

What did you expect to see?

The $GOPATH/src (~/go/src) folder populated with the project's source code, and the binary installed to the bin folder, while honoring the versions specified in go.mod.

What did you see instead?

Only module folders were created.

How does one get the source code, with dependencies, while honoring go.mod?

The one-step install flow of go get was really nice. Will this no longer be the case with Go 1.13 when GO111MODULE=on becomes the default behavior?

More info

We're in an awkward position in our transition to modules. Many people are unable to build Caddy from source without some wrangling.

We've added go.mod and go.sum, removed our vendor folder, and tagged a beta release (in the actual repo, not my test repo here). Unfortunately, modules seem to ignore the beta release tag so all module operations fail unless we explicitly specify the beta version in our commands. So it is not obvious how to test that we've transitioned correctly.

Additionally, without modules (GO111MODULE=off), running go get on Caddy currently fails with:

package github.com/lucas-clemente/quic-go/h2quic: cannot find package "github.com/lucas-clemente/quic-go/h2quic" in any of:
	/usr/local/go/src/github.com/lucas-clemente/quic-go/h2quic (from $GOROOT)
	/Users/matt/go/src/github.com/lucas-clemente/quic-go/h2quic (from $GOPATH

because one of our upstream dependencies have had a breaking change. Our go.mod pins the working version of this dependency, but go get does not honor this (yet?).

But setting GO111MODULE=on also doesn't work because the beta version isn't recognized. The last non-prerelease tag doesn't use modules, so the build also fails.

(As an aside, @v1.0.0-beta1 needs to be specified explicitly, which (temporarily?) breaks our build instructions and isn't obvious... it confused a lot of people including me.)

Even still, if we start over on the test repo but set GO111MODULE=on this time, it does not pull down the source code so we can start developing on it.

If go get is how the GOPATH is populated for development, it should put the source code in it so we can develop with it. If go get becomes a command for a read-only workflow, what is the alternative for "developers" as opposed to just "builders"?

[mholt]

See also https://caddy.community/t/adding-caddy-plugins-with-go-modules-and-latest-caddy-master/5529.

[bcmills]

go get a Go project's source code and install its command (the repo contains both a command and library packages) while respecting module dependencies.

Note that GO111MODULE=on go get does this today. It just unpacks the source code into the module cache instead of GOPATH/src.

There is currently some question as to what to do about replace directives (see #31173), but if you don't need those then you should be set as far as installing binaries.

[bcmills]

But setting GO111MODULE=on also doesn't work because the beta version isn't recognized. The last non-prerelease tag doesn't use modules, so the build also fails.

Yep, that behavior is expected and intended. We want the choice to use an unreleased version (when released alternatives exist) to be explicit.

You can always go get […]@master or go get […]@v1.0.0-beta1. (Or finish up that release and tag it! 🙂)

[bcmills]

what is the alternative for "developers" as opposed to just "builders"?

Generally we're trying to move the go command away from autonomously invoking VCS tools in potentially-nested directories, since that has historically been a source of security vulnerabilities.

The alternative for developers is to run git clone (or your alternative VCS of choice) explicitly. If that alternative doesn't work for you, please let us know more detail so that we can understand the problem and the use-cases.

(See also #18387.)

[bcmills]

(CC @jayconrod, but I don't think we'll be able to make significant changes here until 1.14 at the earliest.)

[mholt]

Thanks @bcmills for taking a quick look at this.

(Sorry for the close+open earlier; my trackpad sometimes does phantom clicks.)

I had always viewed go get as a tool to "go get" source code and populate my GOPATH so that I can work on it.

But now with GO111MODULE=on, and with Go 1.13, I will need to view it almost solely as a dependency manager. (This is a significant change that I think is not obvious.)

I agree it seems like a good idea to not get pre-releases by default... just makes the transition awkward since our beta release is the first to try using modules.

Running git clone is reasonable I suppose, just something new to my workflow which I never had to do much with Go code.

@thepudds gave this 5-star explanation in Slack:

The way it works is if you are inside of a module foo and do go get some/cmd, then some/cmd is treated as a dependency of your current module foo, complete with resolving any shared version constraints with the versions listed in go.mod for foo, updating the go.mod for foo to include a require for some/cmd, and storing the code for some/cmd in the module cache just like other dependencies of foo.

The way it works if you are outside of a module with GO111MODULE=on is briefly documented here:

https://golang.org/doc/go1.12#modules

When GO111MODULE is set to on, the go command now supports module-aware operations outside of a module directory, provided that those operations do not need to resolve import paths relative to the current directory or explicitly edit the go.mod file. Commands such as go get, go list, and go mod download behave as if in a module with initially-empty requirements. In this mode, go env GOMOD reports the system's null device (/dev/null or NUL).

In other words, in that case, if you do cd /tmp; go get some/cmd, then some/cmd is still treated "as if" some/cmd is a dependency of a temporary and otherwise empty module, complete with storing the code for some/cmd in the modules cache along with dependencies of other modules.

So now I think I have something I can tell the community, many of whom are currently having trouble building Caddy from source:

1. git clone https://github.com/mholt/caddy.git
2. cd caddy
3. git checkout <version> ⬅️ optional, but necessary if wanting a specific version
4. go build or go install ⬅️ will get all dependencies as specified by go.mod

Thank you for the help, coming to understand this.

I'll leave the issue open since being able to consolidate those steps into one command (like it has always been before) would be convenient, especially if one (like me! 😄) wants to continue to use a traditional GOPATH. In other words, being able to go get source code as before, but with respect to go.mod, would be great.

One last question: in the future, GOPATH is still where module dependencies will be stored (read-only); will it still be recommended and good practice to store working copies of repos in GOPATH? I ask because I really like its structure, but if it messes with go tooling + modules, I should try to avoid that...

[jayconrod]

One last question: in the future, GOPATH is still where module dependencies will be stored (read-only); will it still be recommended and good practice to store working copies of repos in GOPATH? I ask because I really like its structure, but if it messes with go tooling + modules, I should try to avoid that...

$GOPATH/src is still a good place to keep working copies of repositories. I'd recommend it to keep code organized. However, since the Go command won't look for dependencies there, avoid writing tools or scripts that require this convention.

[krasi-georgiev]

@bcmills can you explain in more details why was this behavior changed?

I really liked the simple instructions to build a binary

go get github.com/mholt/caddy
cd $GOPATH/src/github.com/mholt/caddy
go build cmd/caddy

didn't need to think what is my PWD and didn't need to create all parent folders etc.

[jnjackins]

@krasi-georgiev @bcmills in addition to downloading source code (which has been discussed here already), go get of a main package would by default install to $GOPATH/bin, so instructions for someone to download and run your tool were simply:

$ go get github.com/foo/cmd/bar
$ $GOPATH/bin/bar

Or if $GOPATH/bin is in one's $PATH, simply

$ go get github.com/foo/cmd/bar
$ bar

In Go 1.13 I really miss this functionality. Now I can't do this, or tell someone else to do it, without likely damaging some unrelated go.mod file.

In other words, we used to have a great tool for downloading and installing source code and binaries, now we have a tool that might do that in a sane way depending on the context, or it might instead update module dependency requirements (and install the binary to $GOPATH/bin, which doesn't seem to be the right thing to do in this context).

It would be great if these two very separate use cases were not both crammed into the same command go get.

[jnjackins]

For example a quick twitter search shows lots of examples of people giving instructions to install their tool via go get: https://twitter.com/search?q=%22go%20get%22%20%23golang&src=typed_query&f=live

I don't think updating a go.mod file is what is intended in any of those cases. Asking people to set GO111MODULE a certain way to get the right behavior feels like a kludge.

[jayconrod]

@jnjackins #30515 is the issue for that.

[dsoprea]

Waiting on this. Having to specifically clone all of my projects to the right location in the tree in order to have the replaces all make sense in all actively developed modules just adds extra setup/accounting to development. I appreciate modules in all of their advantages, but it's backwards that this is no longer an automatic part of the workflow. Before, everything was self organizing.

This somewhat reminds me of "automatically"- versus "manually"-installed packages in Ubuntu. The sourcecode for the project(s) passed as arguments to go-get will be brought down as source under $GOPATH/src though their dependencies (if a module) will be established under pkg/. If the dependencies are not modules then just bring those down as source as well. This should maintain the existing ethos of which stuff should be under pkg/ and which should be under src/. This probably oversimplifies the requirements and their ramifications, but any progress in this direction would be good.