Google Cloud client libraries broke with protobuf 4.23.x version.

[Xiroo]

Apache Airflow version

Other Airflow 2 version (please specify below)

What happened

In airflow version 2.6.2 and 2.6.3, there's a constrain like following.
In 2.6.2
protobuf==4.23.2
In 2.6.3
protobuf==4.23.4

But protobuf with version above 3.20.x, it return failure log like following.
I think it is known issue also in the protobuf project

I suggest fixing the constrain about protobuf with version 3.20.1

Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging/__init__.py", line 18, in <module>
    from google.cloud.logging_v2 import __version__
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/__init__.py", line 25, in <module>
    from google.cloud.logging_v2.client import Client
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/client.py", line 25, in <module>
    from google.cloud.logging_v2._helpers import _add_defaults_to_filter
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/_helpers.py", line 25, in <module>
    from google.cloud.logging_v2.entries import LogEntry
  File "/usr/local/lib/python3.10/site-packages/google/cloud/logging_v2/entries.py", line 31, in <module>
    import google.cloud.audit.audit_log_pb2  # noqa: F401
  File "/usr/local/lib/python3.10/site-packages/google/cloud/audit/audit_log_pb2.py", line 62, in <module>
    _descriptor.FieldDescriptor(
  File "/usr/local/lib/python3.10/site-packages/google/protobuf/descriptor.py", line 560, in __new__
    _message.Message._CheckCalledFromGeneratedFile()
TypeError: Descriptors cannot not be created directly.
If this call came from a _pb2.py file, your generated code is out of date and must be regenerated with protoc >= 3.19.0.
If you cannot immediately regenerate your protos, some other possible workarounds are:
 1. Downgrade the protobuf package to 3.20.x or lower.
 2. Set PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION=python (but this will use pure-Python parsing and will be much slower).

More information: https://developers.google.com/protocol-buffers/docs/news/2022-05-06#python-updates

What you think should happen instead

No response

How to reproduce

using 'from airflow.providers.google.common.hooks.base_google import GoogleBaseHook' with protobuf above 3.20.x

Operating System

macOS 13.0

Versions of Apache Airflow Providers

apache-airflow-providers-google==10.2.0

Deployment

Official Apache Airflow Helm Chart

Deployment details

No response

Anything else

No response

Are you willing to submit PR?

• Yes I am willing to submit a PR!

Code of Conduct

• I agree to follow this project's Code of Conduct

[potiuk]

No. Google client libraries deliberately moved to Protobuf 4. Protobuf 3 is end of life already so there is no way we are going to set it in constraints. If this is a problem, then this is a problem with cloud logging library compiled with protobuf that are already end of life.

We have no control over 3rd-party libraries like google logging. It the version that has been fixed in our constraints had protobuf3 -compiled code, then it's the problem of theirs not ours.

The good news is that Airflow mechanism of constraint is prepared and gives yout ways to fix it.

You are free to upgrade those dependncies that have some features that might be limiting you. You can definitely try to upgrade to latest version of google provider and especially to the latest google logging library 3..6.0 that has been released 14th of July in the hopes it will work. In fact it is already tested and runs in main so it should be quite safe to upgrade it (though of course you should test if it works in your case, ideally on staging environment of yours). Just follow the docs of ours - where we explain how to install and upgrade airflow depenendencies after installing airflow "with constraints" https://airflow.apache.org/docs/apache-airflow/stable/installation/installing-from-pypi.html#managing-providers-separately-from-airflow-core

Or you can also upgrade to 2.7.0 when it is out (likely in the next few weeks) if you are willing to wait - we are automatically upgrading the dependencies of ours to latest "compatible versions" and every new relase of airflow will have upgraded dependencies - so if you prefer to base on what community provides, wait for 2.7.0.

Those are basicaly two choices you have:

• if you want to rely on the set of dependencies that community releases - you just upgrade to latest version of Airflow with all the goodies /provider version it comes with - latest and gratest
• or if you need ot update those dependencies out-of-band - follow the instructions we gave to our users so that they can follow them on their own and selectively update what they need.

converting to discussion if more is needed.