[Python] Release 4.21.0 broke multiple Google Cloud client libraries ("TypeError: Descriptors cannot not be created directly.")

[lgruen]

Example error for the google-cloud-logging==3.1.1 library using protobuf==4.21.0:

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging/__init__.py", line 18, in <module>
    from google.cloud.logging_v2 import __version__
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/__init__.py", line 25, in <module>
    from google.cloud.logging_v2.client import Client
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/client.py", line 25, in <module>
    from google.cloud.logging_v2._helpers import _add_defaults_to_filter
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/_helpers.py", line 25, in <module>
    from google.cloud.logging_v2.entries import LogEntry
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/entries.py", line 31, in <module>
    import google.cloud.audit.audit_log_pb2  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/google/cloud/audit/audit_log_pb2.py", line 62, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

Note that this works fine with the previous protobuf==3.20.1 release.

To reproduce:

docker run -it python:3.10 /bin/bash
# Within the Docker container:
pip3 install google-cloud-logging==3.1.1 protobuf==4.21.0
python3 -c "import google.cloud.logging"

[chelseas]

Same issue when using onnx to convert network formats.

[JanuszL]

Same for the TensorFlow 1.15.5.

[gpmahanta]

Same issue for the google-cloud-datastore as well.

[hagen00]

Upgrading the following packages to latest fixed it for me. Not sure which packages actually needed upgrading.

google-api-core==2.8.0
google-auth==2.6.6
google-cloud-core==2.3.0
google-cloud-storage==2.3.0
google-resumable-media==2.3.3
googleapis-common-protos==1.56.1

[stephanie-gillespie]

I've set protobuf==3.20.1 and it resolved the issue

[peterdudfield]

I've set protobuf==3.20.1 and it resolved the issue

Thanks, this also worked for me

[gpmahanta]

I've set protobuf==3.20.1 and it resolved the issue

yes, this thing works but this is not a permanent fix.

Many libraries like google-cloud-datastore are having the dependency on protobuf >= 3.19.0 and most of the platforms like Databricks are already pre-installed with protobuf == 3.17.2 by default which is causing an issue in various platforms and frameworks.

[EduMorenoJ]

setting protobuf==3.20.x or setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python works

[mike-luabase]

setting protobuf==3.20.x or setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python works

Note that PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python will use pure-Python parsing and will be much slower

[jacobg]

@mike-luabase I got this error deploying to Cloud Functions. Is that due to needing to set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python, or due to other libraries not compatible with the new protobuf?

[vitas-pm]

Since the traceback points to

  File "/usr/local/lib/python3.10/site-packages/google/cloud/audit/audit_log_pb2.py", line 62, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

"/usr/local/lib/python3.10/site-packages/google/cloud/audit/audit_log_pb2.py"

Am I wrong to assume that the error comes from Google's own _pb2.py file that has not been regenerated with protoc >= 3.19.0?
The file in their repository hasn't been updated since last year.
https://github.com/googleapis/python-audit-log/blob/main/google/cloud/audit/audit_log_pb2.py

So for now there is nothing we can do but wait and use protobuf==3.20.1 if we want to use google cloud logging.

[mike-luabase]

@mike-luabase I got this error deploying to Cloud Functions. Is that due to needing to set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python, or due to other libraries not compatible with the new protobuf?

Try adding

protobuf==3.20.1

to your requirements, if that fixes it, don't set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python

[EduMorenoJ]

setting protobuf==3.20.x or setting PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python works

Note that PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python will use pure-Python parsing and will be much slower

True, I would just set it if freezing the protobuf version does not work

[FreeBee34]

I would assume that the Google Cloud client libraries have tests on protobuff to ensure that new releases dont break them. I am wondering how this happened

[roomo7time]

4.21.0 is broken for torch tensorboard as well. As mentioned above, easy fix is to uninstall it and install protobuf==3.20.1.

[RafaelFazzolino]

I've set protobuf==3.20.0 and it resolved the issue here

[shsab]

Any permanent fix for this?

[TBBle]

The permanent fix is to regenerate affected _pb2.py files with protoc 3.19 or later, as mentioned in the initial post. (If you are not the owner of the affected files, you'll need to file a bug with the relevant project to do that.)

[gchacaltana]

I've set protobuf==3.20.1 and it resolved the issue

It work!