proposal: x/website: have endpoint to query for the latest bugfix version

[mvdan]

Background

#34864 covers documenting the endpoint better. Right now, it's documented at https://godoc.org/golang.org/x/website/internal/dl. Listing the versions is very useful, so that endpoint is great.

However, there is one common operation that's not covered by the endpoint: showing the latest Go patch release for a major version like 1.12 or 1.13. This is useful to quickly download and install such a latest version, or to use it for a line like docker run golang:$(<query> 1.13.x) ..., and so on.

CI systems like Travis or GitHub Actions allow you to specify 1.13.x as a Go version in their configs. Internally, they query /dl?mode=json, and figure out what the latest version is. That's fine, because that software is pretty complex anyway, so a bit of JSON parsing is not terrible.

However, this gets harder when one is writing shell script one-liners, or portable scripts in general. There's just no way to do it in POSIX Shell or even Bash, let alone shells for other systems.

/VERSION?m=text almost satisfies this purpose, but it only shows the latest single release (not 1.12.x, 1.11.x, etc), and it sometimes lags behind even days after a newer release has gone out.

Proposed solution

Given that /VERSION is a static file and not an endpoint, I'm going to suggest that we extend the existing /dl endpoint instead. But this choice can be changed.

We don't want the result to be JSON, for the reasons laid out before. So the first step is adding ?mode=text alongside ?mode=json. Similar to /VERSION?m=text, it simply prints one version name per line, and nothing else. This is already useful by itself; one could figure out the absolute latest version via curl .../dl?mode=text | sed 1q, for example.

To support querying for the latest bugfix version for a particular "major" release like 1.12, we'd add another parameter latest: /dl?mode=text&latest=1.12.x. Given a string ${version}.x, it returns the latest bugfix release for that version. The version must be in the format ${number}.{number}, and the suffix .x must be present, to avoid ambiguity with 1.12 (actually 1.12.0).

Queries like /dl?mode=json&latest=1.12.x should also work, doing what you'd imagine. Instead of returning one line with just the version name, you'd get a JSON element with the structured data for the release in question.

/cc @dmitshur @myitcv @andybons

[mvdan]

Another option would be to rely more on the shell, and simply add mode=text. Then, the equivalent of curl .../dl?mode=text&latest=1.12.x would be something like curl /dl?mode=text | sed -n '/1\.12\./{p;q}'. However, I didn't include this suggestion because regexes are easy to mess up, and portability remains very difficult.

[dmitshur]

Thanks for filing this.

/VERSION?m=text almost satisfies this purpose, but it only shows the latest single release (not 1.12.x, 1.11.x, etc), and it sometimes lags behind even days after a newer release has gone out.

I agree the /VERSION endpoint is not a great long term solution. It can reliably answer the question of "what VERSION file does the golang.org website serve", but that cannot be reliably correlated with what versions of Go are released or tagged.

Since you haven't discussed it explicitly in the original issue, I'll copy what I wrote in #36886 (comment):

If you're looking for more up-to-date Go release information, I suggest using either the https://golang.org/dl/?mode=json endpoint, which is updated as soon as release downloads are available, or the maintner API (see usage example here), which is updated as soon as releases are cut (the tag is published).

I want to ask what you think about relying on the maintner API for this? Its ListGoReleases endpoint seems to return exactly the information you're looking for. The questions are:

• whether making an unauthenticated gRPC call is considered "convenient" enough for the needs you stated in the original issue
• whether the expectation of stability of the API based on the domain name is good enough (the golang.org vs maintner.golang.org domains may be interpreted differently)
• whether including a new minor release as soon as it is tagged is what you're looking for, or whether you want to know about it only when that minor release is completed (i.e., release artifacts such as downloads are published)

It's still an option to expose the same information from the golang.org/dl?mode=json API, which this issue is about investigating, but I wanted to ask a few of those questions first.

[mvdan]

whether making an unauthenticated gRPC call is considered "convenient" enough

I want to make use of this in scripts, so it should be plan HTTP+plaintext, just like the existing endpoint with mode=text. So a gRPC client won't do, I'm afraid.

whether the expectation of stability of the API based on the domain name is good enough

I'd prefer this to live under golang.org, simply because that's also where one downloads the releases from. It's also a bit more convenient (i.e. shorter) to use. Not a big problem, but if what we want is to consolidate all of this into the maintainer API, we should just deprecate the existing /dl REST endpoint.

Note that, as said before, the replacement needs to support REST and plaintext endpoints too, otherwise we'd be losing valuable features.

whether including a new minor release as soon as it is tagged is what you're looking for

I imagine a large amount of the use cases (including mine) involve using the release, i.e. downloading it. So yeah, the feature should show the latest fully released version.

[mvdan]

I haven't gotten much feedback, so I'm turning this into a proposal in hopes that it goes through the formal process soon. I did write the original issue almost like a small proposal, anyway.

[rsc]

It's worth noting that https://golang.org/dl/?mode=json has the information.

This also requires a developer to change their CI scripts for upgrading the latest Go release.

I guess the question is: do you want your CI system to break automatically the moment we release a new version of Go (if the CI system is fetching "latest"), or do you want to control when the update happens? In general we've encouraged the latter and discouraged references like "latest".

Given that "latest" is possible already with the JSON (and shell scripts can use jq), and that maybe "latest" shouldn't be made trivial to discourage these kinds of blind updates, maybe things are just right as is.

[Copied from #38150. Since this is the older issue, let's put the discussion here.]

[mvdan]

Thanks for your reply. Since your reply was written for the duplicate issue, I think some points need clarifying.

In general we've encouraged the latter and discouraged references like "latest".

In general, I agree. But note that I propose allowing to query the latest 1.13.x too, which is something that lots of CI systems support as well. This is a very useful feature for open source projects, since one wants to support a set of major Go versions. Testing against the oldest 1.13.0 might work, but that's not realistic in terms of what people are actually running, and might be buggy. Having to update to the latest bugfix version manually is a lot of unnecessary churn - having to update the version manually every few weeks, instead of every six months.

Given that "latest" is possible already with the JSON (and shell scripts can use jq)

I disagree - see my reasoning in the original proposal.

[rsc]

@mvdan, the original comment above says:

However, this gets harder when one is writing shell script one-liners, or portable scripts in general. There's just no way to do it in POSIX Shell or even Bash, let alone shells for other systems.

But this is not true. Bash and other systems can use jq, or people can use a simple Go program invoked from the shell, maybe even with go run. (Bash doesn't do HTTP requests internally yet does it? I believe this discussion assumes invoking other tools like curl already.)

You said in the comment at the top that CI systems are fine as is, since they can run basic computation even more easily than shell scripts can.

Even after rereading your original comment atop this issue, I'm not sure I understand what the argument is for making resolving 1.13.x easier for bash.

[rsc]

@mvdan, am I misunderstanding something in my previous comment? This seems headed for likely decline but I want to give you a chance to reply to make sure I'm not missing something.

[mvdan]

Bash and other systems can use jq

It's true that the original example requires some tool to download a file via HTTP (like curl), though that's generally possible to do out of the box on most machines or docker images. Comparatively, jq is pretty niche and rarely installed by default.

or people can use a simple Go program invoked from the shell, maybe even with go run.

I should have clarified that this is generally for CI environments that don't have a Go version installed.

You said in the comment at the top that CI systems are fine as is, since they can run basic computation even more easily than shell scripts can.

Yes and no. Some CI systems like GitHub Actions and Travis wrote their own code to figure out what the latest version for Go 1.14.x is. I can use that, yes, but then I don't have that feature if I use some other CI system, or if I run my own CI. I could write this software myself, depending on jq, python, Go, or some other tool that can process json, but it feels like I shouldn't need to reinvent the wheel here :)

I'm not sure I understand what the argument is for making resolving 1.13.x easier for bash.

That's what I tried to answer in #36898 (comment). A short version of that is - people find it useful for testing on CI.

Here's a perhaps simpler counter-proposal. Only add a ?mode=text, which was the first half of my proposal here. The equivalent of the current:

$ curl -s 'https://golang.org/dl/?mode=json&include=all' | jq -r '.[].version'
go1.14.2
go1.13.10
go1.14.1
go1.14
go1.13.9
go1.13.8
[...]

That way, in mostly-portable shell, one can get the latest 1.14.x by simply doing a sed '/go1\.14\./q'. And we don't explicitly support syntax like 1.14.x, nor do we encourage it.

[dmitshur]

I should have clarified that this is generally for CI environments that don't have a Go version installed.

I think this is a good goal. I filed issue #38773 related to this. Perhaps it will be easier to evaluate any proposed changes to the https://golang.org/dl/?mode=json API by looking at CI environments where it's currently difficult to use.

[rsc]

First, we really need to document the current mode=json endpoint.
Second, that seems like enough for this hypothetical CI system that has curl but not jq.
It's easy enough to pull out what you want:

$ curl 'https://golang.org/dl/?mode=json' | grep '"version"' | sed 1q | awk '{print $2}' | tr -d ',"'
go1.14.2
$

Plenty of other ways too (can do the whole thing with sed or perl too, assuming the CI has at least one of those).

#34864 is the issue for documenting mode=json better.

Based on the discussion above, this issue (a latest-version endpoint or a text mode) sounds like a likely decline.

[rsc]

No change in consensus, so declined.