protobuf recent update broke protoc integration

[pgeler]

Description

new implementation as #20089 of searching protoc actually finds system protoc instead the one build by conan, as a result usage of the protoc leads to error: incompatible with your Protocol Buffer headers. Please #error incompatible with your Protocol Buffer headers. .... linux ubuntu 22.04 with default protoc installed from apt-get

Package and Environment Details

• Package Name/Version: protobuf/3.21.12
• Operating System+version: Linux Ubuntu 22.04
• Compiler+version: Clang 14
• Docker image: conanio/gcc8
• Conan version: conan 1.18.0
• Python version: Python 3.10.12

Conan profile

default

Steps to reproduce

cmake build with dependencies

Logs

Click to expand log

Put your log output here

[SpaceIm]

You have to add protobuf to your build requirements.

[pgeler]

You have to add protobuf to your build requirements.

it is already there

[SpaceIm]

So it should work, please share your conan version, compiler etc, conanfile, CMakeLists and commands invoked (properly fill the issue template in short).

There might be an edge case when user relies on legacy cmake generator, since bin dir of protobuf package is appended to PATH instead of prepended.

[Sfletche132]

Hi I'm seeing the same thing with my build.

import os
from conan import ConanFile


class MyApplication(ConanFile):
    settings = "os", "compiler", "build_type", "arch"

    name = "my_app"
    version = "1.0.0.0"

    # comma-separated list of requirements
    generators = "CMakeDeps"

    keep_imports = True

    default_options = {"grpc/*:cpp_plugin": True,
                       "grpc/*:shared": False,
                       "protobuf/*:shared": False,
                       "boost/*:shared": False,
                       "fakeit/*:integration": "catch"}

    def requirements(self):
        self.requires("boost/1.83.0")
        self.requires("grpc/1.54.3")
        self.requires("openssl/3.1.3")
        self.requires("protobuf/3.21.12")
        self.requires("sqlite3/3.41.1")
        self.requires("spdlog/1.12.0")
        self.requires("nlohmann_json/3.11.2")
        self.requires("eigen/3.4.0")
        self.requires("fmt/10.1.1")
        self.requires("cli11/2.3.2")
        self.requires("zlib/1.3", override=True)
        self.requires("asio/1.28.2")
        # Install libusb on windows, get it from the toolchain otherwise (handled inside device manager)
        if self.settings.arch == "x86_64":
            self.requires("libusb/1.0.25")
        if self.settings.os == "Windows":
            self.requires("sdl/2.0.16")
        # Testing libs
        self.requires("fakeit/2.4.0")
        self.requires("catch2/3.4.0")
        self.requires("fake_clock/1.01")

    def build_requirements(self):
        self.tool_requires("protobuf/3.21.12")
        self.tool_requires("grpc/1.54.3")

    def build(self):
        pass
        # self.output.warn("CC=>{}".format(os.getenv("CC")))

    def imports(self):
        self.copy("license*", dst="licenses", folder=True, ignore_case=True)

    def layout(self):
        # We make the assumption that if the compiler is msvc the
        # CMake generator is multi-config
        multi = True if self.settings.get_safe("compiler") == "msvc" else False
        if multi:
            self.folders.generators = os.path.join("build", "generators")
        else:
            self.folders.generators = os.path.join("build", str(self.settings.build_type), "generators")

I'm using Ninja and invoking protobuf_generate inside my cmakelists.txt

[SpaceIm]

Where is CMakeToolchain?

[Sfletche132]

Where is CMakeToolchain?

Ah yes, I'll try with the toolchain, thanks

[pgeler]

this is the conan file

from conan import ConanFile

my_packages = {
    "asio/1.28.2": {},
    "backward-cpp/1.6": {},
    "benchmark/1.8.2": {},
    "docopt.cpp/0.6.3": {},
    "expected-lite/0.6.3": {},
    "grpc/1.54.3": {},
    "gtest/1.14.0": {},
    "openssl/[>=3.0 <3.1]": {},
    "protobuf/3.21.12": {},
    "range-v3/0.12.0": {},
    "rxcpp/4.1.1": {},
    "spdlog/1.12.0": {},
    "zlib/[>=1.2.13 <2.0]": {},
}

class ConanPkgs(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = "CMakeToolchain", "CMakeDeps"
    requires = list(my_packages.keys())
    default_options = dict(kv for d in my_packages.values() for kv in d.items())

    def configure(self):
        if self.settings.os != "Windows":
            self.options["backward-cpp"].stack_details = "backtrace_symbol"

[SpaceIm]

protobuf is not in build requirements

[pgeler]

sorry what about? "protobuf/3.21.12": {},

protobuf is not in build requirements

[pgeler]

updating conanfile with build requirements did not solve the problem

    def build_requirements(self):
        self.tool_requires("protobuf/3.21.12")
        self.tool_requires("grpc/1.54.3")

[Sfletche132]

I'm using conan through the https://github.com/conan-io/cmake-conan repo. It says the toolchain isn't supported with that method. Is there a workaround to use the above repo with the current protobuf changes?

[pgeler]

Im looking at revisions, so hopefully can use those... while package is fixed. will give more details in an hour or so

[pgeler]

ok I was able to get back to previous revision, just specify it as package reference, also may need to explicitly enable revisions
conan config set general.revisions_enabled=True

"protobuf/3.21.12@#ed2404ae366627005beb2173eab68174"

[pgeler]

@SpaceIm any thoughts?

[SpaceIm]

I don't know what you are doing. You have not shared all elements I asked.

[pgeler]

Does it matter? The problem is that receipt is using incorrect version of protoc and not the one from the package, previously that was not the case, so this is the line which causing the problem

conan-center-index/recipes/protobuf/all/conanfile.py

Line 136 in 5ff3625

if(NOT PROTOC_PROGRAM)

, so it would be great if you check it one more time from perspective of this issue. I can submit the fix, but it can break your intention, which I don't want to, as I have no visibility to original intention.

ps. I don't think it is a productive conversation to point that everybody should just switch over to cmake toolchains because it is working.

[leoparente]

I'm having the same problem as @pgeler and what he mentioned makes sense.

I have protoc 3.12 installed in my system, but for my project I use conan protobuf 3.21. The PR #20089 simply broke my build.. I use legacy cmake with conanfile.txt. I will do a workaround for now, but that is it.

[tgockel]

I'm not that familiar with Conan, but in my case protobuf comes as a transitive dependency through grpc, but I can't figure out how to force a protobuf version to one that works without causing a conflict. For example:

conan_cmake_configure(
  REQUIRES
    "grpc/1.50.1@#01c169a35963fcf8fca63a6f67a7e592"
    "protobuf/3.21.9@#64ce20e1d9ea24f3d6c504015d5f6fa8"
  GENERATORS
    CMakeDeps
    CMakeToolchain
)

This is a bit odd, since grpc/1.50.1@#01c169a35963fcf8fca63a6f67a7e592 was published 2023-06-11, but protobuf/3.21.9@#64ce20e1d9ea24f3d6c504015d5f6fa8 was published 2023-11-21, so clearly I'm missing something.

Trying to specify any other recipe revision leads to:

Graph error
    Version conflict: Conflict between protobuf/3.21.9 and protobuf/3.21.9 in the graph.

---

Conan version 2.0.14. CMake command is just cmake .. -- you don't have to do anything special to show the problem, Protobuf_PROTOC_EXECUTABLE will point to /usr/bin/protoc instead of the Conan location if protobuf-compiler is installed.

[tgockel]

For future readers using cmake-conan: There is no syntax in conanfile.txt that allows you to set a specific version of protobuf and not have it conflict with the Conan-chosen transitive dependency that comes through the grpc package. You must move to conanfile.py.

From the original CMake:

conan_cmake_configure(
  REQUIRES
    grpc/1.50.1
  GENERATORS
    CMakeDeps
    CMakeToolchain
)

Make the conanfile.py in your root:

from conan import ConanFile

class Package(ConanFile):
    settings = "os", "compiler", "build_type", "arch"
    generators = 'CMakeDeps', 'CMakeToolchain'

    def requirements(self):
        self.requires("grpc/1.50.1")
        # grab recipe version from `conan list -r conancenter 'protobuf/${protobuf_version}#*'`
        self.requires("protobuf/3.21.12@#8210c0b1bb46b08ff2814614ec091a9c", override=True)

Then, you can replace the call to conan_cmake_configure with a configure_file:

configure_file(conanfile.py "${CMAKE_CURRENT_BINARY_DIR}/conanfile.py" COPYONLY)
file(REMOVE "${CMAKE_CURRENT_BINARY_DIR}/conanfile.txt")

The file(REMOVE ...) makes sure the old conanfile.txt is cleaned up (you can't have both conanfile.txt and conanfile.py). The use of configure_file will make sure that CMake will properly re-run if conanfile.py changes (this won't happen if you conan_cmake_install(PATH_OR_REFERENCE "${CMAKE_CURRENT_LIST_DIR}" ...)).

[uilianries]

@pgeler Hello! Thank you for reporting! When opening the issue, please, fill the form with all asked information. Otherwise, it will be really hard to reproduce your scenario and confirm your error, plus, work in a hotfix. Could you please update your issue description with all information like: your real profile (conan profile show -pr:h default), the environment (if need to update), and really important: an entire build log with your error.

[ilya-lavrenov]

With updated protobuf recipe OpenVINO cannot be built on WIndows anymore #22244 even for old releases.

[SpaceIm]

see #22542 (comment) and #22244 (comment)

[dpad]

I had the same issue and confirmed that adding protobuf to the build_requirements is the correct fix. It's not enough to have it only in requirements, it needs to be in build_requirements as well because the protoc compiler is used for the build process. I am not sure if this is the exact same problem as what is occurring to some of the users above, but the code below fixed it for me.

The following (based on @SpaceIm 's comment in #22244 (comment)) fixed this issue for me (for both Conan 1 and 2):

@property 
def _is_legacy_one_profile(self): 
    # Returns True if user is running a Conan 1 profile
    return not hasattr(self, "settings_build") 

def build_requirements(self):
    # Include protobuf as a build requirement so that Conan-provided
    # `protoc` is found instead of system-installed one).
    if not self._is_legacy_one_profile: 
        self.tool_requires("protobuf/<host_version>")