Let bazel_deps replace Go deps

[fmeum]

When a Go dependency is brought in both as a bazel_dep and as a
regular Go dependency via e.g. go.mod, the bazel_dep repo should be
used instead of creating a separate go_repository, which would risk
duplicate packages.

This is realized by registering every Bazel module with a
go_deps.from_file tag as providing the Go module of the same version
with the module path given in go.mod.

Gazelle is aware of these tags and generates either a canonical label or
a label using the apparent module repo name (the latter only for the
root module and only if it directly depends on the bazel_dep).

Fixes #1525
Fixes #1543

[fmeum]

@Wyverald @tyler-french Could you review?

I implemented two different approaches (first commit vs. both commits), with one making the association explicit and one deriving it from go_deps.from_file. I prefer the latter, but in case you see issues with this association being made implicitly, we can switch back to the first approach.

[fmeum]

We probably need to introduce some kind of version resolution to ensure that outdated bazel_deps do not override up-to-date go.mod deps.

[fmeum]

I added version resolution.

@tyler-french @seh Could you take a look if time permits?

[seh]

Could you take a look if time permits?

I read it all, but I need to think about both the problem and the solution more before I have anything constructive to add.

[fmeum]

@seh Thank, that's exactly what I'm looking for. Take your time!

[tyler-french]

Tested at Uber

[fmeum]

I rebased onto master and pushed a new commit that adds support for the generalized semver format used by Bazel module registries.

I will wait for bazelbuild/bazel-central-registry#674 and add a test based on it before merging.

[fmeum]

@seh I added a more realistic example of how this would be used in the form of circl.

[seh]

I added a more realistic example of how this would be used in the form of circl.

I hadn't seen a module like circl before, where there's an upstream Go module with no "awareness" of Bazel, but then we publish a Bazel module that adds in all the necessary Bazel control files to allow building and testing the library.

Now, coming back to this patch, I understand it better now, in that we're reading all the uses of the go_dep extension's nomination of Go modules from non-root/child Bazel modules and taking them into consideration when choosing the maximum version for each Go module.

The description mentions a "provides" tag on the go_dep extension. Is that implemented?

[fmeum]

@seh Sorry, the PR description was outdated, updated it to the description of the first commit.

[seh]

I hadn't seen a module like circl before, where there's an upstream Go module with no "awareness" of Bazel, but then we publish a Bazel module that adds in all the necessary Bazel control files to allow building and testing the library.

What's not clear to me yet is why one would express such a Bazel module dependency, as opposed to just letting the go.mod file express the dependency and rely on go_deps.from_file to take care of the rest. It seems like expressing the same idea twice, when our goal with the project is to eliminate as much of the redundant Bazel configuration as possible (e.g. getting rid of the need for use_repo with go_deps).

Are we getting more from that Bazel module than what Gazelle would generate on its own by reading the go.mod file?

[fmeum]

Having to use a Bazel module as a Go module should be rare, but there are important exceptions:

When a Go module requires patches beyond Gazelle's capabilities. A go_deps.module_override can fix that, but makes it so that the Go module can no longer be used by non-root Bazel modules. We could have a separate registry for patches similar to what we do to centralize go_deps.gazelle_overrides, but that seems pretty complex to set up while the BCR provides this out of the box.

The particular case of circl is interesting here as it requires adding cc_library targets, but those will eventually have to be loaded from rules_cc, which in turn requires visibility into this Bazel module - go_deps-generated repos don't have this, only bazel_deps can declare additional dependencies in this way.

Another use case is for something like bazelbuild/buildtools, which provides both Starlark rules, Go tools and Go libraries. Most Bazel users will want the first two and thus probably a bazel_dep, but anyone who consumes the Go library wants to do so via go_deps. We should probably ensure that there aren't two different repos floating around that provide the same Go targets but potentially in different versions.

[seh]

there are important exceptions:

Thank you for the detailed explanation. The dependency landscape is more complex than I had considered.

[aaronmondal]

I've just tested this for rules_ll which uses that exact circl dependency as a transitive dependency. We were previously running into an issue where users could not only not import our go tooling, but couldn't import rules_ll at all because the go toolchain misconfiguration would block the entire build.

Keeping patches like the ones required for circl maintained isn't nice if it requires custom deps and overrides. This patch fixes these issues in an IMO very elegant manner.

So this will also fix #1543 in a very satisfactory way ❤️