proposal: cmd/go: go get should support source archives

[iand]

This is a tester to see if there is any support for this idea. If there is then I will write up a more detailed proposal.

Currently the go get command expects to interact with a source control system to check out source code. There is no way to package up a source archive (e.g. tar or zip) and use go get to retrieve this.

This proposal would modify the go tool to look for a source archive according to a to-be-defined scheme when the user types go get package. The go tool would download and unpack the source archive in the correct location. Alternatively the go tool could be modified to keep the archive file and support reading source from the archive directly during a build.

One possible strawman scheme would be to extend the .vcs package path syntax to allow the specification of an archive format, e.g. example.com/code/package.zip. This would be the direct URL of the source archive of the package.

There are a number of advantages to using source archives:

• simpler publishing of packages: just place the archive file on a web server
• removes need for installing support utilities such as git or mercurial
• convenient versioning of packages by adopting a semver naming convention for the archive
• verifiability through pre-existing content hash and signing schemes
• better caching of source downloads
• opportunities for mirroring
• potential for cleaner inclusion with OS package systems (e.g. apt, yum etc)
• smaller download sizes since no need to replicate entire vcs history

[bradfitz]

Dup of #16545

[iand]

As @bradfitz noted, this is suggesting similar functionality to #16545. That proposal got stuck on the behaviour of go get -u, especially in the situation where the downloaded source had been edited.

I think this complexity can be avoided by stating that source archives are immutable. go get -u example.com/code/package-1.0.zip would have no effect if the directory $GOPATH/src/example.com/code/package-1.0 already exists.

To upgrade to a new version of a package, a new package path needs to be used, e.g. go get -u example.com/code/package-1.1.zip would download and extract source files to $GOPATH/src/example.com/code/package-1.1

[bradfitz]

What if package-1.0.zip changed? Or more realistically, if a user put up mycode.zip and tried to use go get -u on it? Would -u be an error if it already existed on disk? Would we save the server's Content-Length & ETag? Where?

[iand]

Changing package-1.0.zip breaks the contract that source archives are immutable. The user would need to intervene and remove the old version manually. This is the same situation that occurs when getting from a vcs where the user has modified the local source and not committed it or the remote source has rebased the user's current commit out of existence. go get should not try and fix these problems.

-u would be a no-op if the directory existed. The source is assumed to be immutable so it cannot be updated if it already exists. This is similar (I think) to the behaviour of go get in a detached head situation.

I think saving the etag/sha256sum is out of scope of this change. I would expect the go get process to check the sha256sum of the file during download if a convention is defined for the naming of a file containing the sum, e.g. example.com/code/package-1.0.zip.sha256sum

[bradfitz]

Changing package-1.0.zip breaks the contract that source archives are immutable.

I don't think that's a sufficient answer. Users are going to violate it anyway, either on purpose or by accident, and we need an answer to why two users have different results when one ran "go get" at time 1 and the other ran "go get" at time 2, then both ran "go get -u" and things were still different. We deal with enough of those support complaints already. This would make it worse. It's easy enough to say "just go rm -rf $GOPATH/src/foo", but in this case "foo" might be a far away dependency of the thing the user actually cared about.

In other words, we already support the "-u" flag. I don't think it's acceptable to just say that sometimes "-u" doesn't work for certain packages, seemingly randomly to the user.

[iand]

As you pointed out on #16545 we don't want to implement a vcs system. If users want to modify their code in place and reuse the same package path then they should use the pre-existing go get support for common vcs systems. For source archives, versioning is achieved by naming the package paths and assuming the archive is immutable. -u would not download a replacement for a source archive but it would traverse the dependency tree and recursively update imports as normal. -u -v would indicate that the package source is assumed to be up to date.

If there were a convention for locating a remote sha256sum then potentially we could add a way of verifying that the local source has the same hash as the remote but this is only informative.

I don't think it's acceptable to just say that sometimes "-u" doesn't work for certain packages, seemingly randomly to the user.

I don't think it would appear to be at random. The user is typing in a package path that ends with .zip or .tar. This is a signal that something is different and using -v would explain that difference.

Immutability of source archives could be seen as a goal in of itself. The current mechanism for version pinning has confusing behaviour for end users. For example github.com/aktau/github-release uses semver to tag releases (and coincidentally is a tool for doing that). A user can pin a release by checking out a tag, e.g. cd $GOPATH/src/github.com/aktau/github-release; git checkout v0.5

Subsequent usages of go get -u result in a potentially confusing error:

$ go get -u -v github.com/aktau/github-release
github.com/aktau/github-release (download)
# cd /home/iand/src/github.com/aktau/github-release; git pull --ff-only
You are not currently on a branch.
Please specify which branch you want to merge with.
See git-pull(1) for details.

    git pull <remote> <branch>

package github.com/aktau/github-release: exit status 1

This is made more confusing because the user believes they have just switched to the v0.5 "branch". Also -u stops here and doesn't update dependencies of the package.

The source archive approach would retain the semantics of "pinning", guaranteeing that the source is unchanged, but continue to check and update dependencies (which may also be pinned).

[bradfitz]

I don't think it would appear to be at random. The user is typing in a package path that ends with .zip or .tar.

No, I was referring to cases where the .zip is a dependency of something else.

Consider:

User 1

# Time 1:
$ go get github.com/foo/bar
# Time 3:
$ go get -u github.com/foo/bar

User 2

# Time 2:
$ go get github.com/foo/bar
# Time 3:
$ go get -u github.com/foo/bar

Question: do User 1 and User 2 have the same code on disk? Will the github.com/foo/bar package be the same?

With your proposal, you no longer know. Because foo/bar might transitively depend on what you're assuming is an "immutable" package, but only because you're wishfully declaring that. A user could actually put my-foo-bar-dep.zip on an HTTP server and update it, but then Users 1 and 2 above could have different code on disk, even after a go get -u.

[bradfitz]

If you want to proceed with this, please write a design doc. See https://github.com/golang/proposal

[iand]

Thanks, will do.

[ghost]

this will also solve --depth=1 problem (on github) #18423.
e.g go get https://github.com/USER/REPO/zipball/master and go get https://github.com/USER/REPO/tarball/master

[rsc]

On hold for package management, which I expect will have an answer for this problem.

[dmitshur]

I believe this proposal is resolved by now via modules.

A module, which is a collection of Go packages, can be distributed as a zip archive via the module proxy protocol, described here. A Go package with a custom import path can declare that it should be fetched via said module proxy protocol by using the "mod" pseudo-VCS type, documented in the last paragraphs of https://golang.org/cmd/go/#hdr-Remote_import_paths.