x/tools/gopls: once Go 1.23 is released, require Go 1.21+ (forward compatibility) to build gopls

[findleyr]

As described in some detail in #50825, Go version compatibility has long been a pain point for gopls. At the time that issue was written, we hadn't deprecated support for building gopls at any Go versions. Since then, we've successfully dropped support for Go 1.12-1.17, and very soon will drop support for 1.18. These deprecations have for the most part gone smoothly, and we haven't heard about users getting stranded as a result.

Nevertheless, gopls's nominal support for building with the past 4 Go versions continues to cause friction, as outlined in #50825. With Go 1.21's forward compatibility, we look forward to the day when gopls' minimum supported Go version is 1.21, so that we can rely on forward compatibility to build gopls with the latest Go toolchain.

Per the support window of 4 trailing Go versions, that day would occur upon the release of Go 1.24. This issue proposes to move that day up one release cycle, to the release of Go 1.23 this August.

The rationale is as follows:

• The original support window of 4 trailing Go versions was related to the limited availability of recent Go versions in various operating system distributions and cloud providers. Since then the situation has improved significantly, and most release pipelines stay pretty up to date.
• Better support for language versions in the Go toolchain (go list, go/types, etc.) has made it easier to use the latest version of Go in your development environment while targeting deployment with older Go versions.
• Upon the release of Go 1.22, our survey results from VS Code indicated that 91% of our users were on Go 1.20, Go 1.21, or Go 1.22. Translating this to the release of Go 1.23, this suggests that 91% of users would be on Go 1.21 or later. That's a very large majority.
• VS Code support for managing Go and gopls installations has improved significantly (and we can do more before 1.23 if necessary).
• We have plans for standard library contributions in 1.23 that we want all gopls users to benefit from.

This last point is crucial: one of our major goals for 2024 is improved refactoring, and a foundation of that will be improvements to the AST, parser, and type checker (for example fixing #20744). Additionally, with type parameters on aliases (#46477) likely to land in Go 1.23, we are doing a lot of work (in #64581) to handle the new representation of Aliases added to go/types in Go 1.22. It would be a lot of additional effort (and arguably infeasible) to land these improvements with Go 1.23 while simultaneously continuing to support gopls built with older Go versions.

So, in summary, I think requiring Go 1.21+ to build gopls (via downloading the 1.23 toolchain) will save us a lot of resources and allow us to deliver a more reliable and robust product, and there seems to be little downside to dropping support for Go 1.20 at the time 1.23 is released.

EDIT: here are the actions to close this bug, assuming there are no objections:

• update the version support documentation at https://github.com/golang/tools/blob/master/gopls/README.md
• update the version compatibility table in VS Code; gopls@v0.16.0 will be the last gopls version to support both Go 1.19 and Go 1.20.
• start warning users who build gopls@v0.16.0 with Go 1.19 or Go 1.20
• Once go1.23rc1 is released, update gopls/go.mod with go 1.23rc1, and officially drop compatibility for older Go versions.
• make a decision about what to do for CI coverage. It seems OK to leave one legacy LUCI builders, since they will exercise Go command integration with older Go versions

CC @golang/tools-team

[pjweinb]

Telemetry since Jan 1 gives the same results. The percentages for Go versions seen using gopls: devel 5 go1.19 3 go1.20 13 go1.21 71 go1.22 8

…

On Fri, Feb 23, 2024 at 7:06 PM findleyr ***@***.***> wrote: As described in some detail in #50825 <#50825>, Go version compatibility has long been a pain point for gopls. At the time that issue was written, we hadn't deprecated support for building gopls at any Go versions. Since then, we've successfully dropped support for Go 1.12-1.17, and very soon <https://github.com/golang/go/milestone/326> will drop support for 1.18. These deprecations have for the most part gone smoothly, and we haven't heard about users getting stranded as a result. Nevertheless, gopls's nominal support <https://github.com/golang/tools/tree/master/gopls#support-policy> for building with the past 4 Go versions continues to cause friction, as outlined in #50825 <#50825>. With Go 1.21's forward compatibility <https://go.dev/blog/toolchain>, we look forward to the day when gopls' minimum supported Go version is 1.21, so that we can rely on forward compatibility to build gopls with the latest Go toolchain. Per the support window of 4 trailing Go versions, that day would occur upon the release of Go 1.24. This issue proposes to move that day up one release cycle, to the release of Go 1.23 this August. The rationale is as follows: - The original support window of 4 trailing Go versions was related to the limited availability of recent Go versions in various operating system distributions and cloud providers. Since then the situation has improved significantly, and most release pipelines stay pretty up to date. - Better support for language versions in the Go toolchain (go list, go/types <https://pkg.go.dev/go/types>, etc.) has made it easier to use the latest version of Go in your development environment while targeting deployment with older Go versions. - Upon the release of Go 1.22, our survey results from VS Code indicated that 91% of our users were on Go 1.20, Go 1.21, or Go 1.22. Translating this to the release of Go 1.23, this suggests that 91% of users would be on Go 1.21 or later. That's a very large majority. - VS Code support for managing Go and gopls installations has improved significantly (and we can do more before 1.23 if necessary). - *We have plans for standard library contributions in 1.23 that we want all gopls users to benefit from*. This last point is crucial: one of out major goals for 2024 is improved refactoring, and a foundation of that will be improvements to the AST, parser, and type checker (for example fixing #20744 <#20744>). Additionally, with type parameters on aliases (#46477 <#46477>) likely to land in Go 1.23, we are doing a *lot* of work (in #64581 <#64581>) to handle the new representation of Aliases added to go/types in Go 1.22. It would be a lot of additional effort (and arguably infeasible) to land these improvements with Go 1.23 while simultaneously continuing to support gopls built with older Go versions. So, in summary, I think requiring Go 1.21+ to build gopls (via downloading the 1.23 toolchain) will save us a lot of resources and allow us to deliver a more reliable and robust product, and there seems to be little downside to dropping support for Go 1.20 at the time 1.23 is released. CC @golang/tools-team <https://github.com/orgs/golang/teams/tools-team> — Reply to this email directly, view it on GitHub <#65917>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABJIAI57QEXFHJRV3A4MNXTYVEVJLAVCNFSM6AAAAABDXQQLNWVHI2DSMVQWIX3LMV43ASLTON2WKOZSGE2TCOJXGUYDENA> . You are receiving this because you are on a team that was mentioned.Message ID: ***@***.***>

[mvdan]

I fully agree with this. I still hear some users talking about their wish for gopls to be faster or more robust, but I have yet to hear about any pain relating to older Go versions.

Perhaps the first release to require Go 1.21 or later could point users to the last gopls version supporting older Go versions, which could be useful as a temporary fallback.

[findleyr]

I fully agree with this. I still hear some users talking about their wish for gopls to be faster or more robust, but I have yet to hear about any pain relating to older Go versions.

@mvdan thanks for that evidence. I think we may hear about some pain related to this, if we really are requiring 9% of users to take action if they want to upgrade, but I think that inconvenience is well offset by the improved robustness and team efficiency resulting from needing to support building with only the latest Go version.

Perhaps the first release to require Go 1.21 or later could point users to the last gopls version supporting older Go versions, which could be useful as a temporary fallback.

We'll definitely do that. We also have a practice of warning folks interactively from the (N-1)st gopls version when their Go version will be dropped by the Nth gopls version.

[gopherbot]

Change https://go.dev/cl/578039 mentions this issue: gopls: use static hooks rather than optional hooks

[gopherbot]

Change https://go.dev/cl/588763 mentions this issue: gopls: warn about Go 1.19 and Go 1.20

[findleyr]

v0.16.0 tasks are now complete on this issue. Moving to the v0.17.0 milestone to finish.

[findleyr]

With respect to CI coverage: we decided to continue maintaining legacy LUCI builders for the last 3 go versions (not 4, as before). This narrows our support window slightly for Go command integration, but data indicate only a small fraction of users are on go version N-3. This also means that once 1.23 is released, we can rely on toolchain upgrades to build gopls (thanks to https://crrev.com/c/5589446), and get integration testing with older Go versions for ~free: since gopls integration tests run on a temp directory, they will integrate with the local toolchain go command. We can simply drop the 1.20 builder.

[gopherbot]

Change https://go.dev/cl/593682 mentions this issue: x/tools: update go.mod to 1.22 and gopls to go1.23

[gopherbot]

Change https://go.dev/cl/593683 mentions this issue: x/tools: assume go1.22 and simplify

[dylan-bourque]

For an alternative point of view, some development orgs (like mine) are not able to immediately move to new releases of Go right away due to external constraints (regulatory/compliance restrictions, "conservative" leadership, etc.), including explicitly setting GOTOOLCHAIN=local to (mostly) disable automatic toolchain updates.

We do have some engineers who adopt new versions immediately, sometimes even during RC stages, to do verification testing against our codebase, but for the majority of our engineers it will be some time before they are using Go 1.23. We are probably about evenly distributed between 1.21.x and 1.22.x today. There will also be a delay for those running 1.21.x today moving to 1.22.x once 1.23 is released.

With this change, most of our engineers will be unable to go install golang.org/x/tools/gopls@latest (which is what VS Code does internally to update gopls IIRC) because they will be running an "old" version of Go (1.22.x or 1.21.x) even though that version is fully supported by Go itself. This will continue to be the case until/unless everyone updates to 1.23.x.

While I understand the engineering trade-offs related to supporting prior versions of Go, a change that requires all users update to the latest release of Go in order to use gopls seems like an undesirable user experience.

[findleyr]

Thank you for sharing this, @dylan-bourque (and for the discussion in Slack). CC @rsc as I think he would be interested in this perspective.

A point of confusion for me is that when installing gopls@latest you are intentionally getting the latest version of gopls and its dependencies (which includes some limited third party dependencies, such as staticcheck). I don't see why this is fundamentally different from updating the Go toolchain. I'm not being rhetorical: is there something I'm missing that's different? Is the concern that the toolchain is distributed as binary? I understand that organizations may have concerns about upgrading Go in their production workloads, but those concerns don't seem to apply to developer tools. Put differently, what would the policy be if we distributed gopls as a binary? Would you only use older versions of gopls? If so, you can still use older versions of gopls with older Go versions.

We expected that the largest barrier to building with the most recent version of Go was the pain of managing multiple go versions, and that this barrier was mostly eliminated with toolchain upgrades.

To repeat some of what we discussed in Slack: we are not choosing to use the latest version of Go out of a principle that we should only support the latest version. We are choosing this path because we think it leads to the best outcome for our users. While some support burden is alleviated by not having to support goN-3 and goN-2 anymore, the largest win for us as gopls maintainers comes when we don't have to support goN-1, because that means we only ever have to support one version of go/parser and go/types. We really felt this pain recently when adding support for the new gotypesalias=1 behavior.

Furthermore, only supporting building with goN means that if we want to make improvements in the standard library, we take advantage of those improvements in 2-6 months (depending on where we are in the Go dev cycle), rather than 8-12 months if supporting the last two versions. This fundamentally changes the way we approach some problems.

There are also other second order benefits to only supporting the latest Go version. For example, some of the value gopls brings to the Go team is simply being a large and complex project, written in Go, to inform our understanding of the language. If we're stuck targeting an old Go version, we lose a lot of the value of our experience. I personally observed this with generics: while I was very involved with the generics implementation in go/types, it wasn't until within the last year that I could actually use generics in my day to day work on gopls.

So, in summary, while I wouldn't be fundamentally opposed to only advancing gopls's go version in its go.mod file to goN-1, we'd need evidence that the benefit to users of this more conservative approach offsets the benefits I described above.

[mvdan]

With this change, most of our engineers will be unable to go install golang.org/x/tools/gopls@latest (which is what VS Code does internally to update gopls IIRC) because they will be running an "old" version of Go (1.22.x or 1.21.x) even though that version is fully supported by Go itself. This will continue to be the case until/unless everyone updates to 1.23.x.

It seems to me like IDEs like VS Code should be adjusted to no longer blindly install gopls@latest if the user is on an older Go major version (e.g. Go 1.21.x) and isn't using automatic toolchain upgrades (either because they're not yet available to that older version, or because the user set GOTOOLCHAIN=local).

@findleyr In such a scenario, what should the IDE do to figure out what is the newest gopls version it can safely install without forcing the user to upgrade their Go version? The IDE should show a warning in any case, but I suspect that allowing the user to install a slightly older gopls version is better than refusing to continue. From reading https://github.com/golang/tools/releases I can see that gopls v0.16.x is the highest that a user stuck with Go 1.21.x as a build version can install, but there's nowhere to programmatically obtain that data.

Perhaps you could publish a JSON file in a well-known place, to map the latest minor version of gopls that each Go build version can install without toolchain upgrades, like:

{
    "go1.22": "v0.16",
    "go1.21": "v0.16",
    "go1.20": "v0.16",
    "go1.19": "v0.16",
    "go1.18": "v0.15",
    [...]
}

[findleyr]

@mvdan you're right, we've been copying the version compatibility table to VS Code, but it would be better if it were programmatically available somewhere.

For example, we could publish a tiny submodule consisting of just a JSON table, and perhaps a small binary that outputs the version to install (considering GOTOOLCHAIN), or installs gopls, or both. CC @hyangah: Hana, I know VS Code has essentially this in its upgrade logic (which is written in Go). Should we move that to a common location so that it can be used by other editors?

[mvdan]

Minor adjustment to my last suggestion: one can't go install golang.org/x/tools/gopls@v0.16 to automatically select the latest v0.16.x bugfix release, so I think the JSON file must contain the latest full semver version, and needs to be adjusted whenever any minor or bugfix release is done.

and perhaps a small binary that outputs the version to install (considering GOTOOLCHAIN), or installs gopls, or both.

I was thinking something similar: that, even with the JSON file above and some documentation, the logic is non-trivial and having to write it from scratch is likely to cause bugs. A "gopls bootstrap" command sounds reasonable to me, akin to https://pkg.go.dev/golang.org/dl, and it could only require an ancient version of Go.

[hyangah]

The gopls wants to depend on the latest version of go standard library. IMO that's not different from gopls choosing to upgrade its other 3rd party dependencies, except that the only way to use the latest go standard library is to use the latest toolchain.

With this change, most of our engineers will be unable to go install golang.org/x/tools/gopls@latest (which is what VS Code does internally to update gopls IIRC) because they will be running an "old" version of Go (1.22.x or 1.21.x) even though that version is fully supported by Go itself. This will continue to be the case until/unless everyone updates to 1.23.x.

It seems to me like IDEs like VS Code should be adjusted to no longer blindly install gopls@latest if the user is on an older Go major version (e.g. Go 1.21.x) and isn't using automatic toolchain upgrades (either because they're not yet available to that older version, or because the user set GOTOOLCHAIN=local).

Using another helper binary to pick up the buildable gopls version is definitely an option to consider. But, why is it ok to run the (arbitrary) helper binary, but it is not the official go binary (which are signed, and went through a rigorous release process)? We can also embed the version list in the extension; the extension already does it for other 3rd party tools like gofumpt, staticcheck, golangci-lint.

But, note that there is a risk of the old gopls version no longer works with the latest editor extensions. We only actively test against the latest gopls versions.

For example, VS Code Go now delegates many features and functionalities to the gopls and plans to move more to gopls. Test discovery, unittest generation, struct tags modifications are some of the tasks in our plan. They will stop working if gopls is old.

I think setting GOTOOLCHAIN=auto for gopls or other tools installation is the best for the VS Code Go extension.

We will improve the VS Code Go extension to provide a useful feedback message when the installation cannot complete.

Can we discuss the case where using GOTOOLCHAIN=auto go install gopls@latest is undesirable? That will help us formulate the feedback message. Some cases and resolution I have in my mind:

• Working in an air-gapped environment: go install won't work anyway. Provide the manual installation instruction.
• Working with the old version of go that is out of the supported version range -> downgrade the vscode go extension and install the compatible version of gopls.