cmd/go: add go doc -json

[paulvollmer]

To build better language support for editors and ide's it could be helpful to have the docs as json.

For example https://golang.org/pkg/strings.json to get the strings package documentation.
Or via terminal by a flag like go doc fmt -format json

Then the docs can be used to generate snippets based on the documentation.

[joshuabezaleel]

Hi @odeke-em , do you have any recommendation on where should I start looking if I am interested in this issue?
Thank you very much! 🙂

[ianlancetaylor]

@joshuabezaleel Note that this is a proposal. It may be accepted, but so far it has not been accepted. You are welcome to look into the issue, but if the proposal is not accepted your work will not be accepted either. Thanks.

[rsc]

There is a go/doc package that lets you get package docs in Go data structures. It seems like that should be enough.

Failing that, gopls clearly has some way to give docs to the editor for helper popups. What does it use? Also, more generally, gopls is the way to "better language support for editors and ide's". So if more is needed, it should be in gopls, not cmd/go or a URL on a web site.

/cc @stamblerre

[stamblerre]

LSP only supports plaintext or markdown for hover (see https://microsoft.github.io/language-server-protocol/specifications/specification-3-15/#textDocument_hover), but gopls has an experimental structured hover format that uses JSON. It could be exposed on the command line via the gopls definition command, which already has a -json flag. The code for this command can be found here.

[rsc]

@stamblerre, how does gopls get the docs in a data structure? Does it just use go/doc for that parsing?

@paulvollmer, have you tried using go/doc? Is there anything missing from what it does?

[paulvollmer]

@rsc i'm using go doc to get a fast way to the docs.
what i'm thinking about is using go doc also in a machine readable format.
for example go doc net/http -json to get the docs of the net/http package as a json output.

But maybe there is already a tool doing that. how are the html pages of golang.org/pkg generated?

[stamblerre]

@rsc: gopls borrows a lot of code from go/doc to convert the documentation to markdown (comment.go) and uses go/doc directly to get the synopsis. The rest of the data structure is constructed using type information.

[rsc]

go/doc already defines a data structure, doc.Package.
And I believe 'go doc' creates one of those using go/doc and then formats it to text.
We could say that -json dumps the doc.Package itself to JSON.
That would let us reuse the API we have already defined in go/doc instead of having to define something new.

@paulvollmer, if you got a doc.Package from go doc -json would that work for your use case? (We may need to create some parallel structures with the go/token and go/ast pieces properly converted, which we could do.) At the least that seems like a plausible path forward and is consistent with the other various -json flags in cmd/go.

Thoughts?

[paulvollmer]

@rsc yes, i think this should provide the machine readable data i had in mind. let me know if i can help working on that task.

[rsc]

Thanks for the response @paulvollmer.
This seems like a likely accept.
If you'd like to work on an implementation, please feel free.
Leaving open for a week for final comments.

[paulvollmer]

👍 I'll look inside the go doc tool and how i can use the doc.Package to marshal the docs data to json.

[rsc]

No change in consensus, so accepting.

[joshuabezaleel]

Hi @jayconrod , is this issue still open and a good one for beginners? Do you have any recommendation on where should I start looking if I am interested in this?
Thank you very much in advance! 🙂

[agnivade]

Hi @joshuabezaleel - good to see you here. Please take a look at Russ' comment here as to what the implementation needs.

It requires some knowledge of the go/doc and its sibling packages. We would want to dump the contents of doc.Package and whatever else is necessary to display the full information.

[gopherbot]

Change https://golang.org/cl/334876 mentions this issue: cmd/doc: add json flag

[urandom2]

good news: (from the cl, probably worth discussing here though)

This change makes a naive initial attempt to add json output to
cmd/doc. Unfortunately most valid, populated go/doc.Package vars cannot
be directly serialised, due to pointer cycles in several *go/ast.Object
fields. The opinionated decision was made to nil out breaking fields.

A more correct, long term solution is to commit to a serialisation
interface, however this seemed out of scope for this change, due to the
large amount of data and complexity contained within go/doc.Package. The
first option would be to define and check in a wire format like
cmd/go/internal/modcmd.editModuleJSON. Alternatively, we can use the
existing go/doc.Package serialisable, but this may be undesirable.

[martinrode]

I would be interested in this too. Our use case is to a auto-generate yaml / json openapi representation of a Go struct which consumes json in a rest api. I am particularly interested in using the comments of struct fields as description for the openapi property description.