Publish releases to Maven Central

[Fleshgrinder]

It would be great if the protoc plugins are published to MavenCentral the same way Google does it for protoc (and others: protoc-gen-grpc-java, protoc-gen-grpc-kotlin, protoc-gen-validate) so that it can be used in Gradle without having to install anything.

[bufdev]

What is your workflow without these being published?

[Fleshgrinder]

This enables you to include them as part of your Gradle workflow without installing any external dependencies. Basically running lint and break check with Gradle.

https://github.com/google/protobuf-gradle-plugin#locate-external-executables

[bufdev]

Yep, but what is your workflow without these being published? ie what would be the major value add here?

[Fleshgrinder]

Well right now I cannot include them at all in my Gradle build system and either have to shell out to something else like Docker or worse, e.g. asking the user to homebrew whatever or start manually downloading binaries and placing them in their path.

Having it published would resolve that. Combine it with Linux, Mac, and Windows builds and even that is automatically solved for my users because the Gradle Protobuf plugin together with Gradle takes care of everything.

Edit: there is also the matter of the up-to-date checks that are performed by Gradle. It can automatically detect whether something changed, or not, and invoke the task, or not. A feature that I would need to implement on my own without Gradle.

[andrewparmet]

I had some trouble getting Buf to work as part of a call to the protobuf-gradle-plugin - linting worked, but I couldn't work out how to check schema backwards compatibility.

Buf can check compatibility with protobuf schema checked into a git repo - but not when it's called as a protoc plugin. Additionally part of the protobuf-gradle-plugin workflow is that it extracts protobuf dependencies to the transient Gradle build folder. When Buf tries to verify backwards compatibility it insists that the protobuf files you specify compile cleanly, so those dependencies must be committed for that check to work, which I didn't want to do. So the git check wasn't feasible and I had to publish an image somewhere instead.

I ended up implementing a small plugin that simply shells out to Docker and publishes images alongside other Maven publications: https://github.com/andrewparmet/buf-gradle-plugin

Works well enough for me since I have Docker installed in my dev and CI environments, but if you come up with something better let me know.

I didn't think of this before just now, but now that the check is done from an image, the lint and compatibility checks can be done from the protoc plugin again and only the image must be a Docker call. Of course without the plugins published to Maven Central or equivalent I'd have to check in the plugin whereas with Docker I can always use latest (for better or worse).

[Fleshgrinder]

@andrewparmet another problem I discovered with the plugins is that I cannot pass any options to them via the Gradle Protobuf plugin. It's some time ago that I tried it, so I don't remember exactly what the problem was. Anyways, I'm going to publish the buf executables to Maven Central under my namespace until this ticket is being picked up so that I can automate things. Docker is not an option because I'd end up with Docker in Docker in CI (e.g. CircleCI).

[andrewparmet]

FWIW I have my plugin calling Docker in GitHub Actions CI and it seems to work fine.

The Protobuf Gradle Plugin doesn't allow plugin options to be passed - it's a deliberate decision: google/protobuf-gradle-plugin#415 (comment)

[Fleshgrinder]

GitHub Actions run on VMs, that's why you have no issue there.

[andrewparmet]

Given an artifact containing an executable it should be possible to adapt the plugin I've written to use it if that's a path you want to pursue, similar to protobuf-gradle-plugin with protoc.

[bjaglin]

sbt-protoc would also benefit from this (which would contribute to the "drop-in replacement" nature of the internal compiler), as it provides a way to load and run the protoc binary from a maven artifact: https://github.com/thesamet/sbt-protoc/blob/439803e2c2b565f4e6fb1f736e095a3ccca88c7c/src/main/scala/sbtprotoc/ProtocPlugin.scala#L275-L290.

[andrewparmet]

Gradle can do this dependency resolution using a custom Ivy repository: https://github.com/andrewparmet/buf-gradle-plugin/blob/main/src/main/kotlin/com/parmet/buf/gradle/BufSupport.kt#L25

buf-gradle-plugin no longer requires Docker and resolves Buf executables as Gradle artifacts. If it looks like it might fit your use case, feel free to try it out and let me know how it goes.

[Fleshgrinder]

I keep forgetting about the Ivy workaround. 😋 The issue is still valid though, because with this we cannot have auto-updates for buf and have to manually update the tool version. Another things that central enforces is that the releases cannot be overwritten or deleted and they must be signed. Both are important features to secure the software supply chain.

[andrewparmet]

I'm now uploading Buf binaries to Maven Central: https://repo1.maven.org/maven2/com/parmet/buf/buf/1.0.1/

[buildbreaker]

@andrewparmet We recently set up maven publishing. I do want to understand how you're using Buf with gradle :) -- are you in our slack?

cc @elliotmjackson

[andrewparmet]

I have now joined the Slack. I've built a Gradle plugin that integrates Buf with either the protobuf-gradle-plugin or as a standalone tool within a Gradle project.