x/pkgsite: add support for type parameters

[findleyr]

As the proposal for go/ast has stabilized (#47781), I think it is time to start adding support for type parameters in pkgsite.

There is one major obstacle for this, which is that App Engine will not support 1.18 until some time after the 1.18 release, and we want to support type parameters well before then.

My thoughts on how to do this:

• Write a tool to maintain copies of stdlib packages in pkgsite. I made a working prototype of this in https://golang.org/cl/322411
• Copy the following packages to pkgsite: go/ast, go/build/constraint, go/format, go/internal/typeparams, go/parser, go/token, go/printer, go/scanner.
• Wire these in to pkgsite, which already uses a modified copy of go/doc.

I tested that this was possible in https://golang.org/cl/321949.

Once this is done, we should be able to sync these copied packages to tip using go generate. Out of the box this should resolve parsing errors for generic code, and we can proceed with adding any additional features that make it easier to browse code with type parameters in pkgsite. After App Engine supports 1.18, we can delete these copies.

I can do this, but will hold of for a week or two to see if anyone has concerns or better ideas.

CC @julieqiu @jba @jamalc

[dmitshur]

I understand the scope of this issue is x/pkgsite, and I don't mean to change that.

I'll point out that a similar issue may apply to x/website, though golang.org serves documentation under a significantly reduced set of use cases as of #44356. It similarly has a directory with ported stdlib packages (https://go.googlesource.com/website/+/refs/heads/master/internal/backport/).

[jba]

I'm in favor. There is an alternative: move pkgsite to Cloud Run. That would probably be more work, and certainly more risk, but I just wanted to put it out there.

One point you don't mention: none of those packages would be able to use 1.18 features (including generics) for a while. And we won't know that they do until we copy them and recompile pkgsite. On the other hand, I don't imagine they'll change much once generics are done.

[findleyr]

One point you don't mention: none of those packages would be able to use 1.18 features

Right, that's a good point. In https://golang.org/cl/321949 I had to add a stub io/fs for this reason.

IIRC @griesemer and I had discussed and thought this was OK for these packages in particular. We also have a history of maintaining ports like x/tools/go/internal/gcimporter, but that's more work.

[earthboundkid]

FWIW, as a user, I'm writing some generic packages against gotip, and I find it really annoying that I can't read HTML formatted docs yet. If this can ship before 1.18, it would be great for me.

[findleyr]

@carlmjohnson this will definitely ship before 1.18.

Since no objections and #47781 has been accepted, I will start soon.

[toothrot]

Re-milestoning, as we will not be blocking the Go 1.18 release on this issue.

[sfllaw]

I'm in favor. There is an alternative: move pkgsite to Cloud Run. That would probably be more work, and certainly more risk, but I just wanted to put it out there.

As someone who runs an internal pkgsite, I have been building the frontend Docker image and have had good experiences. We don’t run on Google Cloud and there’s an argument that pkgsite should be more agnostic when it comes to hosting?

[findleyr]

It looks like @jba will be able to move pkgsite to Cloud Run, which simplifies things significantly. Reassigning.

[gopherbot]

Change https://golang.org/cl/382974 mentions this issue: pkgsite: prepare for go 1.18

[gopherbot]

Change https://golang.org/cl/382976 mentions this issue: internal/fetch: add generics test

[gopherbot]

Change https://golang.org/cl/382975 mentions this issue: internal/godoc/internal/doc: update go go1.18beta1

[gopherbot]

Change https://golang.org/cl/382977 mentions this issue: internal/godoc: make work with generics

[jba]

This is live.

[gmlewis]

@jba - how do we get previously-scanned sites to be updated?
For example, https://pkg.go.dev/github.com/gmlewis/advent-of-code-2021 is still missing all the packages that utilize generics.

[hyangah]

@gmlewis Is it possible to push a new version? I don't think old versions are re-indexed (pkgsites db grew so big), but new ones will be indexed. Since go1.18 generics spec went through a couple of last minute changes recently, I think it's a good idea to push a fresh new one.

[gmlewis]

@gmlewis Is it possible to push a new version? I don't think old versions are re-indexed (pkgsites db grew so big), but new ones will be indexed. Since go1.18 generics spec went through a couple of last minute changes recently, I think it's a good idea to push a fresh new one.

@hyangah - that worked! Thank you very much!

[anacrolix]

Sorry for the confusion, but what is live? Should https://pkg.go.dev/ be working with generic code? Is there a sample URL?

[hyangah]

@anacrolix https://pkg.go.dev/golang.org/x/exp@v0.0.0-20220212023102-3e31098684e2/slices