-
-
Notifications
You must be signed in to change notification settings - Fork 668
Editor and tool integration
NOTE: This is a design document for building editor and tool integration. For instructions on setting up editors for Bazel+Go projects, see Editor setup.
Bazel users should be able to edit Go source with the full support of editors just as users of the Go command (go build
, etc.) are able to.
Many editor features require deep integration with the underlying build system. For example, "Go To Definition" involves discovering which package a file belongs to, then loading type information for that package and its dependencies. Traditionally, this work has been done by small tools like godef which have hard-coded assumptions about the underlying build system that do not apply to Bazel.
The Go Tools Team has built a new framework around the Language Server Protocol which should enable better integration for all editors, regardless of the build system in use. See Rebecca Stambler's GopherCon 2019 talk go pls stop breaking my editor for an overview of this framework.
The new framework consists of the following layers:
- Editor: starts and runs the LSP server as a separate process. Sends messages to LSP server as the user edits a file or runs commands.
- gopls: official LSP server for Go. Implements commands such as "Go To Definition". Tells the editor to display errors and lint warnings. Keeps metadata such as file locations and type information in memory.
- golang.org/x/tools/go/packages: Framework for loading metadata about Go packages. Clients ask for information about packages named by command line arguments understood by the underlying build system.
-
gopackagesdriver: An optional tool invoked by
golang.org/x/tools/go/packages
. If present, calls topackages.Load
will be delegated to this tool. Requests and responses are encoded as JSON, passed over stdin and stdout. If not present,go/packages
uses an internal driver that works withgo list
. -
Build system: The system used to build packages from the source being edited. For example, Bazel, Buck, or
go build
.
This document is a design for a new gopackagesdriver binary that integrates Bazel+rules_go with golang.org/x/tools/go/packages
. Once implemented, tools built on go/packages
(like gopls) will work with Bazel.
The driver must be built and installed in PATH
or named explicitly with the GOPACKAGESDRIVER
environment variable.
The configuration is passed to the driver as a JSON packages.driverRequest
object written on stdin.
// driverRequest is used to provide the portion of Load's Config that is needed by a driver.
type driverRequest struct {
Mode LoadMode `json:"mode"`
// Env specifies the environment the underlying build system should be run in.
Env []string `json:"env"`
// BuildFlags are flags that should be passed to the underlying build system.
BuildFlags []string `json:"build_flags"`
// Tests specifies whether the patterns should also return test packages.
Tests bool `json:"tests"`
// Overlay maps file paths (relative to the driver's working directory) to the byte contents
// of overlay files.
Overlay map[string][]byte `json:"overlay"`
}
The list of targets is passed to the driver as a list of command line arguments in the syntax of the underlying build system. For example, a target might be @com_github_pkg_errors//:go_default_library
for Bazel.
The command line arguments may also include queries that start with file=
. For such a query, the driver should return the package that includes the given source file.
The driver should print results on stdout as a JSON packages.driverResponse
object.
// driverResponse contains the results for a driver query.
type driverResponse struct {
// Sizes, if not nil, is the types.Sizes to use when type checking.
Sizes *types.StdSizes
// Roots is the set of package IDs that make up the root packages.
// We have to encode this separately because when we encode a single package
// we cannot know if it is one of the roots as that requires knowledge of the
// graph it is part of.
Roots []string `json:",omitempty"`
// Packages is the full set of packages in the graph.
// The packages are not connected into a graph.
// The Imports if populated will be stubs that only have their ID set.
// Imports will be connected and then type and syntax information added in a
// later pass (see refine).
Packages []*Package
}
The response must include all packages that were matched by command-line arguments. If package dependencies were requested, the response must include transitive dependencies as well (but only directly matched packages should appear in the Roots
list). Note that the ID
field of each Package
is specific to the underlying build system. For Bazel, it will be a Bazel label.
The driver should not exit with a non-zero status unless there was an error that prevented it from collecting data on any packages (for example, bazel
was not installed). Instead, errors should be returned through the Package.Errors
field for each package.
Note that the driver should produce results, even when invoked outside of a Bazel workspace. This may be implemented by recursing into packages.Load
with GOPACKAGESDRIVER
set to off
.
The driver will be a go_binary
at @io_bazel_rules_go//go/tools/gopackagesdriver
.
It will run bazel
with the command line arguments it was invoked with (though special handling will be needed for standard library packages, tests, and file=
queries, see below). Bazel will build one .json file for each package. Each .json file will contain a JSON-serialized partial Package
object. The serialized Package
objects will only use relative (reproducible) paths to source files, generated files, and compiled output files; the driver is responsible for reading source files and filling in details. For rules that build multiple packages, such as go_test
, multiple .json files should be produced.
Go rules don't normally produce .json package files, so these will be built using aspects (the driver will pass an --aspects
option to bazel build
). Several aspects will be provided to produce different sets of outputs, depending on the requested packages.LoadMode
. Only one aspect will be used for any invocation.
-
gopackagesdriver_files
- information forNeedName
andNeedFiles
. The source files will be included in the output in addition to the generated .json. -
gopackagesdriver_export
- additional information forNeedCompiledGoFiles
andNeedExportsFile
. The Go package will be compiled and included in the output. For cgo packages, this will require running cgo and C compiler, too. -
gopackagesdriver_files_deps
,gopackagesdriver_export_deps
- same as the above, but files from transitive dependencies will be included as well.
When the driver runs Bazel, it will use the --build_event_binary_file
option to write build status information to a temporary binary proto file. The format of this file is described by build_event_stream.proto. This tells the driver whether the build was successful and where to find the output files for each package.
Once the build is complete, the driver will read the generated .json files. Relative paths to source files in these JSON files will be converted to absolute paths. The driver will load any additional information requested by the client (for example, imports or type information). It's important this information is loaded by the driver (not the builder) because only the driver has access to an overlay that may change the content of source files. The driver may also exclude fields not requested by the client (though note that go/packages
will also do this).
After processing packages, the driver will print a serialized driverResponse
object on stdout that includes all requested packages and, if requested, their dependencies.
The driver needs to be able to load packages from the standard library, which will be prepared differently from non-standard packages. Standard library packages may be named on the command like using their import paths (e.g., fmt
, encoding/json
). All packages in the standard library may be named with the special pattern std
, which is equivalent to the target @io_bazel_rules_go//:stdlib
.
When the driver is asked to build standard packages, it will build @io_bazel_rules_go//:stdlib
with one of the aspects listed above. This will produce a .zip file containing .json package files for all packages in the standard library. The driver will extract packages it needs from this .zip file. The same .zip file may be used for all modes. Source files will be included in the output and will not be stored in the .zip file.
When the driver receives a file=
query command line argument, the driver will attempt to convert the argument into a Bazel package name, then it will build all
targets in that package with one of the above aspects. Packages that don't include the queried file will be excluded from the output.
Editors may query information about packages that have unsaved changes in source files. The unsaved changes are represented as an "overlay", which is a map from absolute paths to file contents (map[string][]byte
).
When the driver parses files to load imports or type information, it should use overlay contents if present instead of source file contents. Overlays may change package names and build tags, so in general, the driver will need to filter out source files that don't belong in a package (the .json package files will include sources that may be excluded).
Overlays may include files that don't exist on disk yet, possibly in directories that have Go targets or build files. We probably can't support this. In Bazel, a Go package is defined by a go_library
target (or something compatible). If there's no target, there's no package. The target can't be built if all source files are not present.
Each go_test
rule represents up to three packages in addition to the library under test: an internal test package, an external test package, and a generated main package. The driver may only return information about these packages if the tests
flag is set in the driverRequest
.
These packages will have the same Bazel label and import path. To distinguish them in the output, we'll add a suffix to each package's ID
field.
Package | Suffix |
---|---|
library under test | none |
internal test | " [internal test]" |
external test | " [external test]" |
test main | " [test main]" |
Note that go list
has a different suffix scheme. Tools should treat the ID
field as opaque, but since there's not a good way to distinguish test packages, we might need to do something similar. For a package path p
, go list
uses the following suffixes:
Package | Suffix |
---|---|
library under test | none |
internal test | " [p.test]" |
external test | "_test [p.test]" |
test main | ".test" |
.json package files will be produced by one of the aspects defined alongside the driver. For each target, the aspect may format the JSON and create the file with ctx.actions.write
. It should not be necessary for an aspect to create any new Bazel actions.
-
gopackagesdriver_files
- can produce JSON fromGoSource
provider. Not recursive. -
gopackagesdriver_export
- can produce JSON fromGoArchive
provider. Not recursive. -
gopackagesdriver_*_deps
- same as above, but recursive ondeps
andembed
edges.
The aspects above should have a special case for the standard library. We'll produce a .zip file of .json package files for each package in the standard library. A Bazel action will be used to produce this file, and it will be implemented as a new subcommand in @go_sdk//:builder
. The subcommand will run go list -json std
and will transform the output as needed to produce the .zip file. The same .zip file may be used in all modes. Test packages won't be included.
The aspects above should should support all packages produced by go_test
. Currently, only the GoArchive
provider for the generated test main package is exposed in the API. go_test
should be modified to point to GoArchive
or GoArchiveData
for the internal and external test archives more prominently.
A .json file should be generated for each package. During analysis, we don't know which source files belong to which package, so we should include all test source files in both the internal and external packages. The driver will need to sort this out. The driver must be responsible for this because the overlay may contain changes to package names and build constraints.
The driver may be tested using a go_bazel_test
rule, which creates a small test workspace. The test will depend on a wrapper binary that passes its arguments to bazel run @io_bazel_rules_go//go/tools/gopackagesdriver
within the workspace. The test will set GOPACKAGESDRIVER
to the wrapper binary, then it will invoke golang.org/x/tools/go/packages.Load
in various modes and compare the results against expected results.
We need to test:
- Regular targets and standard library targets.
- Various
LoadMode
values fromgo/packages
. All aspects should be tested. Dependencies should be included or not. Metadata should be added or excluded as necessary. - With and without nogo. Nogo errors should be included in the
Errors
field. - cgo.
- Internal and external tests.
- Overlays should be able to modify type information for existing files.
- Overlays should be able to switch a test source file from internal to external or vice versa.
- Various build modes
- This will likely be rewritten in the future, but we should start with cross-compilation using
--platforms
. Different source files should be included in the output, depending on build constraints.
- This will likely be rewritten in the future, but we should start with cross-compilation using
-
file=
query should work if file exists. - Error conditions
- Missing targets reported as list errors.
- Missing source files reported as list errors.
- Syntax errors in source files reported as parse errors.
- Type errors in source files reported as type errors.
- Other compile errors (including nogo errors) reported as unknown errors.
- When invoked outside a Bazel workspace,
golang.org/x/tools/go/packages
returns results as if no driver were present.
None of the gopackagesdriver
aspects shall be considered part of the rules_go public API for now. The interface between the driver and rules_go is private and may change without notice. An installed driver may check the version of rules_go to ensure compatibility. Alternatively, we may recommend installing a script that invokes the driver via bazel run
instead of installing the driver itself. This would ensure the driver is always up to date.
We may eventually make these part of the public API in order to support tools that need to use golang.org/x/tools/go/packages
within an action.
- Should gopackagesdriver run Bazel in a separate output base? Bazel only allows one build per output base, so this would unblock builds in the main workspace. However, the additional local cache would consume a lot of space and memory.
- How can actions write file names to generated JSON package files? Absolute paths won't be meaningful for remote or sandboxed actions. We could restrict the actions to being local only or we could write relative paths and absolutize them in the driver.
- Can other tools based on
golang.org/x/tools/go/packages
consume the files produced by the gopackagesdriver output groups in Bazel actions? This would enable more code generation tools. We might need a separate gopackagesdriver for this, but hopefully we can use the same data. - Should metadata that requires parsing source files be produced by the builder or by gopackagesdriver? If it's produced by the builder, it can be cached by Bazel, but we won't be able to include absolute paths. If it's produced by the driver, we may need to do more work locally. The driver will need to parse files in overlays at minimum, so the implementation may be simpler if the driver does most of the work.
- Should an internal test package have a different package path than the package under test?
- Should we produce .json files with aspects or output groups? Aspects let us decouple the new logic from the regular analysis and execution logic. However, they may be a little more difficult for other Bazel actions to integrate with (unless the aspects are part of the public rules_go API).
- Should we include cgo source files except in
gopackagesdriver_files
mode? These files can't be type checked. Probably not.