Python: protobuf==4.21.0rc1 adds dependency on grpcio-tools

[parthea]

What version of protobuf and what language are you using?
Version: 4.21.0rc1
Language: Python

What operating system (Linux, Windows, ...) and version?
Ubuntu 20.04.4 LTS

What runtime / compiler are you using (e.g., python version or gcc version)
Python 3.9.12

python -m grpc_tools.protoc --version
libprotoc 3.19.4

What did you do?
Steps to reproduce the behavior:

1. Run pip install googleapis-common-protos
2. In python shell, run >>> from google.rpc import error_details_pb2
3. Observe that no error occurs
4. Run pip install protobuf==4.21.0rc1 to install the pre-release version of protobuf
5. In python shell, run >>> from google.rpc import error_details_pb2
6. Observe the following error

>>> from google.rpc import error_details_pb2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/google/rpc/error_details_pb2.py", line 52, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
AttributeError: module 'google._upb._message' has no attribute 'Message'. Did you mean: 'CMessage'?

7. Install grpcio-tools==1.44.0 or newer
8. In python shell, run >>> from google.rpc import error_details_pb2
9. Observe that no error occurs
10. Uninstall grpcio-tools
11. In python shell, run >>> from google.rpc import error_details_pb2
12. Observe that no error occurs

What did you expect to see
I expected to run from google.rpc import error_details_pb2 in a python shell successfully after installing both googleapis-common-protos and protobuf==4.21.0rc1

What did you see instead?
I saw this error

>>> from google.rpc import error_details_pb2
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.9/site-packages/google/rpc/error_details_pb2.py", line 52, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.9/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
AttributeError: module 'google._upb._message' has no attribute 'Message'. Did you mean: 'CMessage'?

The strange thing is that installing grpcio-tools solved the issue, however removing grpcio-tools did not bring the issue back.

Make sure you include information that can help us debug (full error message, exception listing, stack trace, logs).

Anything else we should know about your project / environment
I used a clean environment for testing, only installing googleapis-common-protos and protobuf==4.21.0rc1

[haberman]

_message.Message._CheckCalledFromGeneratedFile()

This error indicates that you are using old generated code. The file error_details_pb2.py must have been generated by a protoc prior to 3.19.0.

Per the update at https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates, protobuf 4.21.0 requires that the _pb2.py files are generated from protoc >= 3.19.0

If you regenerate the _pb2.py files it should fix the problem.

The strange thing is that installing grpcio-tools solved the issue, however removing grpcio-tools did not bring the issue back.

My hunch is that installing grpcio-tools downgraded protobuf to 3.x.

[nfelt]

@haberman - since it looks like 4.21.0 final is not yet released, could the protobuf team please consider improving the error message that users get in this case to make it clearer what's broken, prior to the final release?

For example, maybe google._upb._message.Message could be populated with some shim that emits an error message pointing users to https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates or some other indication that the protobuf dep is too new for the generated code to work?

The context is that my team also got a report of this issue: tensorflow/tensorboard#5703 Older releases of our pip package that have generated Python proto files don't have an upper bound on their protobuf runtime dep, which historically hadn't been a problem, but I'm concerned it will become a significant breakage as soon as 4.21.0 is released, and it would help a lot of if the error message was a bit more self-explanatory that downgrading the protobuf dep to the 3.x major version series would at least temporarily resolve the problem.

Edited to add that this will also break a lot of existing released TensorFlow pip packages with the same unclear failure mode, as I think you've seen: tensorflow/tensorflow#56077 (comment)

I am not on the TF team but FWIW, my understanding is that they are blocked on upgrading protoc due to tensorflow/tensorflow#53234 which is very unlikely to be something that can be resolved in a few days, and TF's release cadence almost certainly wouldn't allow rushing out a patch release in that time frame either, so a fix on the TF side seems very challenging given the relatively short notice.

[haberman]

could the protobuf team please consider improving the error message that users get in this case to make it clearer what's broken, prior to the final release?

Yes I think that is a good idea. I had been thinking a similar thing.

[nfelt]

@haberman Thanks for considering, that would be much appreciated 👍

[haberman]

PR: protocolbuffers/upb#617

[xkszltl]

Regarding protobuf "4.21", does that mean the python version will diverge from mainstream version?