cmd/go: allow go-import <meta> tags to specify a branch

[alanconway]

Allow meta tags in 'go get' to specify a branch. Suggested syntax from discussion at https://groups.google.com/d/msg/golang-dev/SW8r9ODYQf0/kqofheawGWYJ

<meta name="go-import" content="qpid.apache.org/proton git https://git-wip-us.apache.org/repos/asf/qpid-proton.git/proton-c/bindings/go branch=go1">

Rationale: there are several reasons why you might want go get to retrieve from a branch that is not "master", "trunk" or the default branch for a repository.

• master/trunk usually used for development, a project may want direct "go get" to get the latest stable release or snapshot on a branch or tag.
• make "experimental" work available from a branch before it is considered stable/mature enough to move to the project's main branch.
• some repos have odd branch naming conventions for historical reasons (e.g. git repos that were converted from SVN and still use "trunk" instead of "master")

[bradfitz]

/cc @adg

[rsc]

I wonder if it should be rev= instead of branch=, so that other non-branch identifiers can be used too. That said, too late for Go 1.6. Need more details. Perhaps the right next step is to write a proposal.

[alanconway]

rev= makes sense to me, released labels could be useful as well as branches.

[elithrar]

I'd be interested in picking this up—which includes writing the proposal—but want to hash out the design a little more and am seeking feedback:

Q. How should the revision/branch/tag support work? Ideally we want to support all three forms but we will need ways to identify which form the go-imports tag is referring to.
Proposed Answer: <meta name="go-import" content="import-prefix vcs repo-root repo-rev"> where repo-rev is of the form branch=$name, rev=$commit or tag=$tag.

Q: What does backward compatibility look like here?
Proposed Answer: Because older versions of the Go tool will not be aware of the additional tag, they will likely (TODO: check this) pull from HEAD instead of the specified rev. This implicit behaviour will be confusing to package consumers as a package author might set up the import URL as getattest.io/pkgname tag=v2, but the consumer won't get this revision and might wonder why their program is breaking or otherwise different.

Should we fail outright here? Rely on // +build go1.7 in the package itself to prevent the behaviour (has other ramifications)? This needs more thought. One solution may be to re-order the go-imports format to force this failure—e.g. import-prefix vcs repo-rev repo-root—at the risk of a confusing error message from earlier versions of go get and other tools.

There are likely other questions here as well, but I wanted to get these down to get the discussion moving.

[alanconway]

Q1. For git I would say rev=xxx (or commit=xxx) is sufficient for repo-rev. The git tools that take a commit will accept a branch, tag, hash or any other string that can name a commit, so I would expect this to do the same (i.e. I can say rev=mybranch, rev=mytag, rev=xxxxcommithash and all will work) I don't know if that is true for other repo types (mercurial, bzr etc.) or if they need the extra info to interpret the string as a branch/tag/etc. Possibly this needs to be different to follow the normal syntax for different repo types?

Q2. Fail outright. Anyone who needs backwards compat will have to set things up to work the old way anyway (HEAD or the magic go1 branch), so there's no point in them using the new feature. If they use the new feature it is safe to assume they have not intentionally set things up to work the old way, and if it appears to work by accident Bad Things will probably happen.

Perhaps a safe way to ensure a clean failure would be use <meta name="go-import-rev">, so old go won't even find the new directive. That would also allow you to set up both "go-import" for old go and "go-import-rev" for new go, e.g. go-import might point to a different repo that has the proper branch cloned as master for backwards compat.

[rsc]

Q1. If you say rev= then that's supposed to work for any code identifier. You don't have to say branch= sometimes or tag= other times, just like you don't have to tell, say, git checkout what kind of argument you are giving it.

Q2. No, older versions of the go command will ignore the tag entirely, resulting in not being able to resolve the reference. That's fine. If you need to specify a branch, then you can't speak to them.

[elithrar]

Thanks for the responses @alanconway and @rsc.

The rev/branch/tag suggestion was primarily for compatibility across VCS', but since git, Mercurial, bazaar and Subversion all just take a revision of any kind it's unnecessary (as Russ points out).

I'll look to push a patch to support an updated go-import tag of the form <meta name="go-import" content="import-prefix vcs repo-root repo-rev"> when I have some free time (moving continents).

[alanconway]

Q2: I believe there will be some set of go versions that does understand <meta name="go-import" but does not understand the new rev= Since rev= implies the author does not want you to pull master it would be bad if they simply ignored the rev= and pulled master anyway. Some kind of failure is called for in that case I think.

[elithrar]

I think my original proposal to just re-order the fields: repo-rev before
the URL - would be sufficient to prevent that.

I'll see how the current tool parses the meta tag when I'm in front of a PC.
On Wed, 30 Dec 2015 at 5:16 AM, alanconway notifications@github.com wrote:

Q2: I believe there will be some set of go versions that does understand <meta
name="go-import" but does not understand the new rev= Since rev= implies
the author does not want you to pull master it would be bad if they
simply ignored the rev= and pulled master anyway. Some kind of failure is
called for in that case I think.

—
Reply to this email directly or view it on GitHub
#10913 (comment).

[rsc]

@alanconway, @elithrar regarding Q2 (again):

Q2. No, older versions of the go command will ignore the tag entirely, resulting in not being able to resolve the reference. That's fine. If you need to specify a branch, then you can't speak to them.

They ignore the meta tag because it has the wrong number of fields. See the source code (src/cmd/go/discovery.go):

    if f := strings.Fields(attrValue(e.Attr, "content")); len(f) == 3 {
        imports = append(imports, metaImport{
            Prefix:   f[0],
            VCS:      f[1],
            RepoRoot: f[2],
        })
    }

[duzy]

You may try editing the HEAD ref. Or just use godev.io or gopkg.in. Where godev.io allows you to select any revision of commit/branch/tag/vX.Y.Z , but gopkg.in is restricted to vX.Y.Z

Examples:

go get godev.io/golang/go.master

Or

go get gopkg.in/golang/go.v1.9

Such revision selection is very useful, unfortunately go tool don't allow such usage. It uses the vendor approach. So things like godev.io and gopkg.in came to help.

[bcmills]

Compare #26964, which proposes to put the branch information in the go.mod file rather than the meta tag.

[bcmills]

Looking at the use-cases again in light of modules:

• master/trunk usually used for development, a project may want direct "go get" to get the latest stable release or snapshot on a branch or tag.

In module mode, if the repository is tagged using v-prefixed semantic versions then go get -u will get the latest stable release. go get with a branch obtains the latest commit on that branch. go get with a tag or commit hash obtains that commit as long as it is reachable from any branch.

• make "experimental" work available from a branch before it is considered stable/mature enough to move to the project's main branch.

Modules support both branches and pre-release tags for this purpose.

• some repos have odd branch naming conventions for historical reasons (e.g. git repos that were converted from SVN and still use "trunk" instead of "master")

This one isn't obvious to me. Is there anything we need to do to make these repos work more smoothly in module mode?

[bcmills]

Is there anything we need to do to make these repos work more smoothly in module mode?

That seems like the only remaining question: if not, then I think we can close this issue in favor of #26964.

[alanconway]

On Wed, Jan 23, 2019 at 11:05 AM Bryan C. Mills ***@***.***> wrote: Is there anything we need to do to make these repos work more smoothly in module mode? That seems like the only remaining question: if not, then I think we can close this issue in favor of #26964 <#26964>.

+1 - updating a .mod file in the repo is a lot easier than finding & changing meta tags on a web page somewhere - it's potentially a whole different set of permissions and tools involved, depending on how the web site is administered.

…

— You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10913 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XizpldhL9kSEKrYMvJdjJpStT5S2ks5vGIhAgaJpZM4EgW9p> .

[zx2c4]

Currently go get resolution uses whatever the default HEAD of git is. This is extremely limited, and has lead to all sorts of horrific hacks, such as gopkg.in having to proxy the git-http protocol itself in order to send a hacked HEAD for go to fetch.

The reasonable and straightforward way of solving this is by allowing go-import meta tags to specify a branch in the git repo it points to. This way various APIs and programmatic path names could be setup to point to different branches.

This old request here remains very valid. Using go.mod is not a solution to the problem faced here. There might be other merits in that approach for other use cases, but it doesn't account for the significant usefulness of being able to programmatically point a go package's name at arbitrary branches, regardless of how somebody has wired up their particular go.mod.

[alanconway]

@zx2c4 is right, silly me - updating a .mod file serves no purpose unless 'go get' can download the .mod file, which requires a revision, which gets back to the original question.

I've been maintaining a project exported using the 'go1' branch for some time now and it works fine - it's just a bit obscure that a branch called "go1" is magically the exported latest-stable code. If it was just called "go-get-go1", and could be a tag as well as a branch, I think I would never have raised this issue.

[bcmills]

If the user's go.mod file requires a valid version, then go get will fetch that version regardless of what branch it is on. If the user wants go get -u to track a specific branch, that is arguably up to that user (that's #26964).

If the user runs go get -u or go get $PKG@latest, then they will receive the version from the most recent release tag regardless of what branch it is on: so all you as a maintainer need to do to point go get -u to a particular branch is to apply your release tags to that branch.

[alanconway]

On Mon, Mar 4, 2019 at 8:40 AM Bryan C. Mills ***@***.***> wrote: If the user's go.mod file requires a valid version, then go get will fetch that version regardless of what branch it is on. If the user wants go get -u to track a specific branch, that is arguably up to that user (that's #26964 <#26964>).

If the user runs go get -u or go get $PKG@latest, then they will receive

the version from the most recent tag regardless of what branch it is on: so all you as a maintainer need to do to point go get -u to a particular branch is to apply your release tags to that branch.

The issue (as I raised it) is about how the package maintainer chooses the code that will be downloaded by default "go get" - the latest stable release. In my project the tip of master is unstable development, releases are tagged and branched - common practice. So "go get" can't take the tip of master by default. I currently merge out to the magic "go1" branch on each release so "go get" can work. It is workable but a pain and breaks go's nice ability to release direct from the development repo without separate packaging steps. If I had a go-only repo things would be simpler, but if the Go tools were just a *teeny bit* more flexible I could get the benefits of their simplicity in a multi-language repo - just tag the release and Go. — You are receiving this because you were mentioned. Reply to this email directly, view it on GitHub <#10913 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XoTqg9PFXTIoMg1XUi0unMWGi-Mnks5vTSJhgaJpZM4EgW9p> .

[bcmills]

The issue (as I raised it) is about how the package maintainer chooses the
code that will be downloaded by default "go get" - the latest stable
release.

In Go 1.13, the default go get will be in module mode — see #30228. (We don't plan to make further improvements to GOPATH mode.)

go get in module mode chooses the latest release tag by default, regardless of the branch on which it appears. So you will indeed be able to “just tag the release and Go.”

[zx2c4]

go get in module mode chooses the latest release tag by default, regardless of the branch on which it appears.

It'd be much preferable to restrict this to the default branch, and then allow the go-import meta tag to specify which branch is pointed at by a particular package name. Otherwise git becomes significantly gimped, as there's no way to put various things in different branches and hope there will be something coherent out of it.

[bcmills]

Not every provider of modules is a git repository: notably, servers can vend module zip files directly, just as module proxies do.

If the packages are in separate subdirectories, you can always put the go.mod file within the subdirectory and include the subdirectory as a prefix on the version tags. But I'm not sure what value you'd get from branches at that point.

See previously #26664.

[zx2c4]

Not every provider of modules is a git repository:

Yes, of course. But that doesn't change the fact that supporting git branches would be very useful.

If the packages are in separate subdirectories, you can always put the go.mod file within the subdirectory and include the subdirectory as a prefix on the version tags.

Sounds like a better use case for branches, so that potentially totally different histories are not intermingled.

[alanconway]

On Mon, Mar 4, 2019 at 9:39 AM Bryan C. Mills ***@***.***> wrote: In Go 1.13, the default go get will be in module mode — see #30228 <#30228>. (We don't plan to make further improvements to GOPATH mode.) go get in module mode chooses the latest release tag by default, regardless of the branch on which it appears. So you will indeed be able to “just tag the release and Go.”

I'm happy with this approach, I like it better than the meta tags. Curious: why "v1.2.3"? We use plain "1.2.3", which seems to be what https://semver.org/ recommends. I quote: "Additional labels for pre-release and build metadata are available as extensions to the MAJOR.MINOR.PATCH format", (e.g. 1.2.3-rc 1.2.3-alpha) but I see no mention of prefixing the version with a "v" to form release labels. You are receiving this because you were mentioned.

…

Reply to this email directly, view it on GitHub <#10913 (comment)>, or mute the thread <https://github.com/notifications/unsubscribe-auth/AHa6XnsSBoiQVt-D2OmrC6E6czkNsrluks5vTTA8gaJpZM4EgW9p> .

[gdamore]

See also #30647 and nanomsg/mangos#78 for some of the grief this causes. The go modules approach, which basically turns some paths into magical (the semver stuff), is breaking without any kind of escape hatch for folks who don't want to adopt modules. Go 1 promise busted. The way to get back is to offer something like proposed here so that it can be possible for existing repos to support semver, vanity names, and both legacy go get and go modules.

[SrslyJosh]

This is a definite usability issue from the developer (not module maintainer) point of view.

While it might be undesirable from a purist standpoint, if I need to fork and patch someone else's module to fix a bug, I want to be able to:

1. Keep my changes in a non-master branch
2. Tell go to use that branch, ideally in the import statement, just by tacking on an "@branchname" or similar

I just want a simple, clear workflow that'll let me unblock myself while waiting for the module maintainer to review my pull request, without having to learn the intricacies of go module release practices.

[bcmills]

@SrslyJosh, you can execute go get some/module@branchname in module mode today. (This feature request is not for that use-case.)

[seankhliao]

The default behaviour of go get (@latest, tracking release versions) seems well understood at this point.
Closing in favor of #26964