Missing upb backend

[NoAnyLove]

Solution to issue cannot be found in the documentation.

• I checked the documentation.

Issue

Protobuf Python has switched to use upb as default backend since 4.21.0, which improves the performance.

The wheels download/install from pypi (e.g., link) include the upb backend (under google/_upb folder), while the conda packages (e.g., link) do not have it.

Can we add the upb backend to conda builds?

Installed packages

N/A

Environment info

N/A

[xylar]

@NoAnyLove, I think this would involve adding upb as its own package on conda-forge. I don't see any evidence that this has been done yet. If this is something you are interested in doing, I think that would be the first step toward including it as a back-end here. See https://github.com/conda-forge/staged-recipes.

While I see how upb would add value, I don't agree that not including it is a bug so I'm going to remove the bug label. I know you don't have many choices when you submit and issue to conda-forge feedstocks so maybe you didn't necessarily think of it as a bug either.

[NoAnyLove]

Unfortunately, I'm not familiar with bazel, not sure if I can create the recipe for it, but I'll take a look when I get time.

Sorry for the confusion, I didn't mean to label it as bug, it's just the default template 😄

[h-vetinari]

Can we add the upb backend to conda builds?

upb is a minimal protobuf implementation; it doesn't have any releases, and gets included in protobuf directly. Since we build the whole protobuf on all platforms, I don't see why the minimal variant would be necessary or even desirable.

If there are functional differences between the different backends, then you should add an upstream issue to expose an option to build upb through CMake.

[h-vetinari]

Ah, I now saw the note in the docs:

There are three separate implementations of Python Protobuf. All of them offer the same API and are thus functionally the same, though they have very different performance characteristics.

The runtime library contains a switching layer that can choose between these backends at runtime. Normally it will choose between them automatically, using priority-ordered list, and skipping any backends that are not available. However you can request a specific backend by setting the PROTOCOL_BUFFERS_PYTHON_IMPLEMENTATION environment variable to one of the
following values:

1. upb: Built on the upb C library, this is a new extension module released in 4.21.0. It offers better performance than any of the previous backends, and it is now the
   default. It is distributed in our PyPI packages, and requires no special installation. The code for this module lives in @upb/python.
2. cpp: This extension module wraps the C++ protobuf library. It is deprecated and is no longer released in our PyPI packages, however it is still used in some legacy cases where apps want to perform zero-copy message sharing between Python and C++. It must be installed separately before it can be used. The code for this module lives in google/protobuf/pyext.
3. python: The pure-Python backend, this does not require any extension module to be present on the system. The code for the pure-Python backend lives in google/protobuf/internal

[h-vetinari]

See also protocolbuffers/protobuf#12927

[h-vetinari]

So the C++ backend now got removed in v26.x, and grpc 1.63 still vendors upb. So now we have exactly the situation that protocolbuffers/protobuf#12927 tried to avoid.