Update to Protobuf 4.x

[neeme-praks-sympower]

Edit by @ejona86: Protobuf has resolved the incompatibility with protobuf-java 4.x. For current status, see my two comments starting at #11015 (comment)

---

Is your feature request related to a problem?

I would like to use the latest released version of Protobuf (4.26.0) but I cannot as gRPC is still using the old version.

Describe the solution you'd like

Update Protobuf dependency and make it work with the new version.

Describe alternatives you've considered

I hoped that the update would be easy (just a version bump) so I tried doing it myself but it resulted in a compilation failure so I'll leave it to you to figure it out.

The failure:

> Task :grpc-protobuf:compileJava FAILED
grpc-java/protobuf/src/main/java/io/grpc/protobuf/StatusProto.java:38: error: cannot access GeneratedMessageV3
          ProtoLiteUtils.metadataMarshaller(com.google.rpc.Status.getDefaultInstance()));
                                                                 ^
  class file for com.google.protobuf.GeneratedMessageV3 not found
grpc-java/protobuf/src/main/java/io/grpc/protobuf/StatusProto.java:171: error: cannot access Builder
        .setCode(status.getCode().value());
        ^
  class file for com.google.protobuf.GeneratedMessageV3$Builder not found
2 errors

[ganadist]

https://protobuf.dev/news/2023-12-05

Also, I'm wondering how this protobuf changes will affect

• gRPC public API changes
• Compatibility with artifacts which were generated by old protobuf/gRPC compilers.

[ejona86]

The protobuf breakage is pretty serious and goes against wisdom like https://jlbp.dev/JLBP-7 . I don't see how the community can absorb it. googleapis libraries are hit even harder than we are, so I think we'd wait until they have an idea how to deal with this (otherwise they couldn't upgrade to newer grpc versions). I don't expect that to be too big of a problem for the community, as I think most protobuf-using projects will be in shock.

I'm in discussions with others, but I'd expect the process to figure this out to take months.

[augi]

After updating to Protobuf 4.26.0, we observe this issue:

java.lang.NoClassDefFoundError: com/google/protobuf/GeneratedMessageV3
	at java.base/java.lang.ClassLoader.defineClass1(Native Method)
	at java.base/java.lang.ClassLoader.defineClass(ClassLoader.java:1017)
	at java.base/java.security.SecureClassLoader.defineClass(SecureClassLoader.java:150)
	at java.base/jdk.internal.loader.BuiltinClassLoader.defineClass(BuiltinClassLoader.java:862)
	at java.base/jdk.internal.loader.BuiltinClassLoader.findClassOnClassPathOrNull(BuiltinClassLoader.java:760)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClassOrNull(BuiltinClassLoader.java:681)
	at java.base/jdk.internal.loader.BuiltinClassLoader.loadClass(BuiltinClassLoader.java:639)
	at java.base/jdk.internal.loader.ClassLoaders$AppClassLoader.loadClass(ClassLoaders.java:188)
	at java.base/java.lang.ClassLoader.loadClass(ClassLoader.java:525)
	at io.grpc.health.v1.HealthGrpc.getCheckMethod(HealthGrpc.java:38)
	at io.grpc.health.v1.HealthGrpc.getServiceDescriptor(HealthGrpc.java:413)
	at io.grpc.health.v1.HealthGrpc.bindService(HealthGrpc.java:350)
	at io.grpc.health.v1.HealthGrpc$HealthImplBase.bindService(HealthGrpc.java:169)

The announcement of the Protobuf release contains this: [Java] The base class for generated messages will be GeneratedMessage, not GeneratedMessageV3.

So I guess this has to be adjusted.

[suztomo]

@neeme-praks-sympower Does any library or dependency require you to upgrade Protobuf runtime version to 4.x?

[be-hase]

It seems that proto-google-common-protos, which grpc-protobuf depends on, is not yet ver4 compliant.
https://github.com/googleapis/sdk-platform-java/tree/main/java-common-protos

[krickert]

Why couldn't grpc just bring back generated message v3? The classes appear almost identical, I suppose it can just extend it?

I tried that myself - it somewhat worked. I got past the generated message v3 error, but I was getting serializaiton errors everywhere. I think they may have changed the serializaiton spec too?

Anyway, I'm looking forward to this fix. I might try this again today.

[neeme-praks-sympower]

@neeme-praks-sympower Does any library or dependency require you to upgrade Protobuf runtime version to 4.x?

No. We just try to keep our dependencies up-to-date.

[suztomo]

@neeme-praks-sympower Thank you.

[ummels]

I've opened an issue at Protobuf for restoring GeneratedMessageV3 as an alias for GeneatedMessage.

[krickert]

I've opened an issue at Protobuf for restoring GeneratedMessageV3 as an alias for GeneatedMessage.

I didn't think you're going to win that though

It was a major upgrade with breaking changes. They deprecated V3 awhile ago, better to fix instead of bringing it back. It'll take some time, but GRPC isn't always caught up to the latest protocol buffers.

[ascopes]

@ejona86 thanks, was asking as on the most recent version of protobuf and gRPC, the introspection service for gRPC-java was still failing due to the use of v3 APIs within protobuf.

[ejona86]

@ascopes, use protobuf-java 4.27.4 or later.

[ascopes]

thanks, shall give it a try.

[dan-lind]

@ascopes, "We're targeting upgrading in 1.67". The schedule for 1.67 is in the milestone (Sept 17th). But you don't have to wait on us. You can upgrade protobuf today in your own app.

Was wondering if there are any updates on this? From what I can tell there is still no update, and it's not targeted for 1.69 either?

[ejona86]

The current plan is "wait and see." We're staying on 3.x until it seems our downstreams are successfully upgrading to 4.x. Basically, we're waiting until it is clear the ecosystem is compatible compatible with 4.x. That is most safely done from the leaves of dependency trees up to the root.

Notably, Google Cloud Java has a separate BOM for protobuf java 3.x, which is identical to the regular BOM except which protobuf-java runtime version it uses. They're watching the uptake of the 4.x version and we're talking with a few larger companies to see if there's common libraries that need fixes before being compatible with 4.x.

Us staying on 3.x shouldn't impact anyone for now. But we do encourage using 4.x when able so that you're more prepared for protobuf-java 5.x. If one of your dependencies isn't compatible with 4.x, please file an issue with them (and if it is a commonly-used library/project, you can share a link to the issue here for our tracking).

[ejona86]

Summary of the situation for projects wondering what they should do:

• Everyone: Replace all references of GeneratedMessage and GeneratedMessageV3 outside of protoc-generated code with Message (or MessageLite, if you care about Android)
• Applications: Try to upgrade to protobuf-java 4.27.4 or later 4.x version by adding an explicit dependency in your project. These versions are compatible with 3.x generated code using protoc 21.8 from Oct 2022. You can depend on protobuf-java 4.x even when your dependencies depend on protobuf-java 3.x. If you have problems with libraries being incompatible with 4.x, file bugs
• Libraries:
  • For protobuf-java 3.x and 4.x support: Use protoc 25.x for your generated code, if at all possible. Versions as old as 21.8 seem to work, but haven't seen as much testing. Depend on protobuf-java 3.x
  • For protobuf-java 4.x and 5.x support: drop protobuf-java 3.x support by upgrading to the latest protoc. You could upgrade to any protoc version after 26, which will all be compatible with protobuf-java 5.x. But latest is best
• Understand protobuf versioning:
  • protoc Y is the same release as protobuf-java X.Y (the protobuf version becomes protobuf-java's minor version).
  • protoc Y generated code requires at least protobuf-java X.Y. When running protoc yourself, there's no benefit to using an older protoc though; just match the versions between protoc and protobuf-java

[toddbaert]

I just wanted to confirm that for our library, @ejona86 's suggestion above worked. We are using protoc v25.5 (via the protobuf-maven-plugin and com.google.protobuf:protoc:3.25.5). Applications using 3.25+ of protobuf-java or 4+ of protobuf-java both work without any issues.

One thing we had to do was specifically pin the version of protobuf-java we use in our build (which generates sources) to 3.25.5 using dependencyManagement to keep things compatible for consumers.

[jitrapon]

I would like to second that @ejona86's suggestion to explicitly declare com.google.protobuf:protobuf-java to be the same version as the com.google.protobuf:protoc works for me in my Gradle project as well. I was importing an older version of the protobuf-java as a transitive dependency and it was on v3, which was probably why the protoc generated classes/methods that don't exist in the older protobuf-java.

[infa-kvaibhav]

@ejona86 Still after using protobuf-java 4.27.4 , seems like makeExtensionsImmutable() is removed from GeneratedMessageV3 which is causing method not found with oci-java-sdk-addons-sasl:3.55.1. Same method is persent in protobuf-java 3.x.

[ejona86]

@infa-kvaibhav, makeExtensionsImmutable() is from ~21.7 and older which is too-old of generated code. The library needs to be rebuilt with protoc 25 to support both 3.x and 4.x. See protocolbuffers/protobuf#17247 (comment)