Split googleapis into per-language modules

[fmeum]

Introduces new googleapis-* modules that contain the per-language module dependencies needed to support the individual rules used by the main googleapis module. This allows BCR modules to declare dependencies on language-specific rules in googleapis without requiring that module to depend on every possible language ruleset, which isn't sustainable.

This results in a breaking change for root modules that previously used the switched_rules extensions. These modules now need to explicitly declare deps on the googleapis-* flavors they need, but will be guided through the process by error messages.

In the future, new flavors can be added in a backwards compatible way.

[bazel-io]

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (googleapis-cc, googleapis-go, googleapis-java, googleapis-python, googleapis) have been updated in this PR. Please review the changes.

[bazel-io]

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (googleapis-cc, googleapis-go, googleapis-grpc-cc, googleapis-grpc-java, googleapis-java, googleapis-python, googleapis) have been updated in this PR. Please review the changes.

[bazel-io]

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (googleapis-cc, googleapis-go, googleapis-grpc-java, googleapis-java, googleapis-python, googleapis) have been updated in this PR. Please review the changes.

[fmeum]

TODO: Replace with a better source after #3474.

[mering]

Couldn't we just allow url to be not set when overlay is used?

[fmeum]

We could, but that would require raising the minimum Bazel version to at least 8.1.0 (the earliest we can ship this).

[mering]

Another thing which just came to my mind: Uploading new versions will become much more work with this approach. This is especially relevant since we don't have active maintainers for this package and are even further away from release automation. So it will become less likely that people contribute version updates.

[fmeum]

We should avoid making updates more complicated, but I don't see why this would make it so. Instead of the patches, there are now a few overlay files, but that's the only change I see. The "flavor" modules only need new versions if the targets generated in googleapis change in a fundamental way. Do you see other downsides to this approach?

I'm thinking of adding a tool to the repo that creates a new version of an existing module by symlinking over files that haven't changed and replacing the old version with the new version in module files, URLs and prefixes. That should make it very easy to update googleapis, regardless of which approach is used. What do you think about that?

[mering]

We should avoid making updates more complicated, but I don't see why this would make it so. Instead of the patches, there are now a few overlay files, but that's the only change I see. The "flavor" modules only need new versions if the targets generated in googleapis change in a fundamental way. Do you see other downsides to this approach?

I raised everything which was on my mind for now in #3437 and here.

I'm thinking of adding a tool to the repo that creates a new version of an existing module by symlinking over files that haven't changed and replacing the old version with the new version in module files, URLs and prefixes. That should make it very easy to update googleapis, regardless of which approach is used. What do you think about that?

It would be great if such tool would also handle patches (at least in cases where it applies with somewhat larger fuzzing for moved lines and skip already applied patches). It could create a temporary git repo for this for example (git init) and delete the .git folder on cleanup.

[fmeum]

@meteorcloudy @Wyverald Friendly ping

[meteorcloudy]

Looks good to me, but wait for @Wyverald to confirm this is the way to go.

[Wyverald]

Sorry about the delay -- this slipped my radar and I've been distracted by other work. I'll take a look later today.

[SanjayVas]

While I really like this approach to the extensions, I'm actually wondering if it's better to make a bigger break to simplify things. What I mean by that is not having googleapis provide language-specific targets and only provide proto_library targets.

The main reason I see for having googleapis provide the language-specific targets at all is to ensure that there's no duplication of generated code. Language-specific proto library rules should all be implemented with aspects though, which inherently avoids the duplication problem.

The downside of course is that there's a much bigger change for downstream users. Additionally, it would be best if the above change could be made on googleapis itself rather than having someone maintain that as patches to BCR.

[fmeum]

@SanjayVas As much as I would like that, it seems almost impossible without cooperation from upstream and as we have learned, they don't care about Bazel at all.

[Wyverald]

Hey, I know I promised to take a look today, but this is quite a big change and I'd like to understand the context better before making a call -- I'll continue to read this (and @mering's previous PR) tomorrow.

[Wyverald]

I've finally caught up, I think. Overall, I really like this approach -- it strikes the perfect balance between working with what we have (the googleapis source repo) and ensuring future adaptability. Kudos! :D

A few more points I was thinking of:

• First off, just a comment: I had been working on a draft PR that had a similar idea, except that the "registration" of a rule was done by the ruleset itself. This is obviously not great in hindsight, especially compared to this PR :P rules_java shouldn't need to depend on googleapis. On the other hand, this PR suffers a similar (minor) problem to that PR, in that there's no "strict deps" for registration: if my dep has a dependency on googleapis-java, I get Java rules for free, which means that a version upgrade of my dep could break me. I don't think this is a solvable problem though, so no sweat.
• There's a fair amount of code in this PR, all in overlay files. I think this is the sort of stuff that would benefit from being in an actual repo (easier to review, etc). Probably bazel-contrib could host this. The repo would contain sources for all googleapis-* modules, and even a googleapis-registry module that contains nothing but the register extension definition. Then we wouldn't need any overlay in the BCR (except for googleapis's MODULE.bazel file itself).
• One of the requests from the protobuf team was support for a "nodep" bazel_dep; that is, specification for the minimal required version of a certain module, without actually introducing a dep on it (so the requesting module cannot use it, even if the dep is present in the dep graph). This would make it more feasible for them to split up the protobuf module. Similarly, it could probably help smooth the version requirements here: the "central" googleapis (or googleapis-registry) module could have a "nodep" spec for all googleapis-* flavors, so that nobody ever has to worry about version skew. Obviously no action required for this PR, but I'm interested to hear your thoughts.

[Wyverald]

should we increment the compatibility level here? IIUC this is a change that breaks almost everyone who uses the old googleapis ("almost" because those who only use proto_library targets should be unaffected), so upping the compatibility level should be a win.

[fmeum]

My rule of thumb for this is:

• Does it only break the root module (rather than transitive dependencies, that is, BCR modules)? If so, asking the root module to migrate to the new version typically doesn't require significantly more effort than setting up a multiple_version_override.
• Would it be reasonable to have a project depend on two different versions of the module? If not, then multiple_version_override will impact maintainability of the root module.

In this particular case, the answers to these questions are "yes (and migrating just requires a few bazel_deps, so you won't need to touch files beyond MODULE.bazel)" and "hell no (a single version of googleapis is already troublesome enough)".

[fmeum]

On the other hand, this PR suffers a similar (minor) problem to that PR, in that there's no "strict deps" for registration

Yes, this is a caveat that is relevant, but I also don't see a way to avoid this.

There's a fair amount of code in this PR, all in overlay files. I think this is the sort of stuff that would benefit from being in an actual repo (easier to review, etc).

I'm still not sure what I prefer, but this approach has an important drawback: It gets much harder to contribute new googleapis versions. Since upstream is not going to support this, we would need a dedicated bazel-contrib repo that hosts the new files. But as a result, external contributors would need to send a PR against that repo, get it reviewed and merged. This makes the maintainers of that new repo a bottleneck to contributions. As I'm not an active user of googleapis and upstream may break us at any time, I wouldn't really want to be in that position.

If we keep the sources in the BCR, the users of the module are in more direct control over the module (assuming we always have sufficiently many maintainers). Since the overlays should change only very rarely (essentially someone adds a new googleapis-foo module) and new versions without changes to them can symlink everything over, I don't find this situation too bad for the BCR experience.

One of the requests from the protobuf team was support for a "nodep" bazel_dep; that is, specification for the minimal required version of a certain module, without actually introducing a dep on it (so the requesting module cannot use it, even if the dep is present in the dep graph).

Yes, that's the missing piece to making this design maintainable in the long run - without it, we wouldn't have a way to force new googleapis-foo versions.

[Wyverald]

I'm still not sure what I prefer, but this approach has an important drawback: It gets much harder to contribute new googleapis versions.

To be clear, the googleapis module would stay (mostly) the same -- its sources will still come from https://github.com/googleapis/googleapis. It's just all the googleapis-* flavor modules, and a central googleapis-registry module that contains the module extension, that will come from the bazel-contrib repo. googleapis's MODULE.bazel file in BCR would then use the extension from googleapis-registry.

[bazel-io]

Hello @bazelbuild/bcr-maintainers, modules without existing maintainers (googleapis-cc, googleapis-go, googleapis-grpc-cc, googleapis-grpc-java, googleapis-java, googleapis-python, googleapis-rules-registry, googleapis) have been updated in this PR.
Please review the changes. You can view a diff against the previous version in the "Generate module diff" check.

[fmeum]

This is now stacked on #3786, which adds the separate modules.