[question] Can an editable package be build using only cmake commands

[mattangus]

What is your question?

Let's say I have two libraries, test_core_lib and test_use_lib. They are set up in the standard way (using conan new):

File structure

test_core_lib
├── CMakeLists.txt
├── conanfile.py
├── include
│   └── test_core_lib.h
├── src
│   └── test_core_lib.cpp
└── test_package
    ├── CMakeLists.txt
    ├── conanfile.py
    └── src
        └── example.cpp
test_use_lib
├── CMakeLists.txt
├── conanfile.py
├── include
│   └── test_use_lib.h
├── src
│   └── test_use_lib.cpp
└── test_package
    ├── CMakeLists.txt
    ├── conanfile.py
    └── src
        └── example.cpp

test_use_lib depends on test_core_lib, I can build these together by running

cd test_core_lib && conan editable add . && cd ..
cd test_use_lib && conan install . && conan build . --build=editable

This works just great! Now I want to use vscode and the cmake extension. Which boils down to running

cd test_core_lib && conan editable add . && cd ..
cd test_use_lib && conan install .
cmake --preset conan-release
cmake --build --preset conan-release

This results in the following error

error

Preset CMake variables:

  CMAKE_BUILD_TYPE="Release"
  CMAKE_POLICY_DEFAULT_CMP0091="NEW"
  CMAKE_TOOLCHAIN_FILE:FILEPATH="test_use_lib/build/Release/generators/conan_toolchain.cmake"

-- Using Conan toolchain: test_use_lib/build/Release/generators/conan_toolchain.cmake
-- Conan toolchain: Setting CMAKE_POSITION_INDEPENDENT_CODE=ON (options.fPIC)
-- Conan toolchain: C++ Standard 17 with extensions ON
-- Conan toolchain: Setting BUILD_SHARED_LIBS = OFF
-- The CXX compiler identification is GNU 11.4.0
-- Detecting CXX compiler ABI info
-- Detecting CXX compiler ABI info - done
-- Check for working CXX compiler: /usr/bin/c++ - skipped
-- Detecting CXX compile features
-- Detecting CXX compile features - done
-- Conan: Target declared 'test_core_lib::test_core_lib'
CMake Error at build/Release/generators/cmakedeps_macros.cmake:67 (message):
  Library 'test_core_lib' not found in package.  If 'test_core_lib' is a
  system library, declare it with 'cpp_info.system_libs' property
Call Stack (most recent call first):
  build/Release/generators/test_core_lib-Target-release.cmake:24 (conan_package_library_targets)
  build/Release/generators/test_core_libTargets.cmake:24 (include)
  build/Release/generators/test_core_lib-config.cmake:16 (include)
  CMakeLists.txt:4 (find_package)


-- Configuring incomplete, errors occurred!

Is there any way to get cmake to trigger the build of the upstream editable packages?

Related to this. Is there a way to only configure the first time? I.e. something like

    def build(self):
        cmake = CMake(self)
        if not cmake.has_config():
            cmake.configure()
        cmake.build()

The reason being that I have a project with several dependencies. The cmake configure stage for these can take a bit of time and I don't want to have to run that every time I build.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @mattangus

Thanks for your question.

It seems you are hitting one limitation of the cmake for multiple editable packages without a workspace.

Is there any way to get cmake to trigger the build of the upstream editable packages?

The way that editable works is that each editable package is still an independent project. They could actually be using different build systems. So one CMake project of one package is completely unaware of the existence of the other editable one, it access it via find_package() in the same way it would find it if it was in the Conan cache. The only difference is the location of the dependency, but it is still an "external" dependency to the project, and as such, it is impossible for CMake to launch the build of the dependency.

Related to this. Is there a way to only configure the first time? I.e. something like

You might be able to move the cmake.configure() step to the generate() method instead of doing it in the build() method, so it runs at conan install time instead, if that is what you mean.

The reason being that I have a project with several dependencies. The cmake configure stage for these can take a bit of time and I don't want to have to run that every time I build.

Well, I think the conan build .. --build=editable will still run the generate() methods of the editable packages, so that might not really save time. But in any case, those configure steps should be really fast once the projects have already been configured, isn't it the case?

It seems that you are looking for the workspace feature. We had something in Conan 1.X, but it was very basic and incomplete, so it was removed in Conan 2. We have already started to resume work on the workspace for Conan 2, but this is quite a challenging feature, so it might take a bit.

In the meantime you might be able to try some custom solution, like introducing something in CMakeLists, as execute_process, or writing a root CMakeLists that orchestrates the build of the different editables.

Please let me know if this clarifies a bit the issue.

[mattangus]

That all makes sense thanks! I read a little bit about the workspaces but didn't go too far because it's not in 2.x yet.

It seems like there is a new feature of cmake (since 3.24) OVERRIDE_FIND_PACKAGE that allows FetchContent_MakeAvailable to be called when find_package is called (see this). Could it be possible to detect when the dependency is in editable mode, then use FetchContent_MakeAvailable?

It may only work nicely cmake dependencies though, since the fallback would be to set

BUILD_COMMAND "conan build ..."
CONFIGURE_COMMAND "conan install ..."
INSTALL_COMMAND "conan ..."
TEST_COMMAND "conan ..."

If you think that is a viable route, would you accept a PR doing something like this?

[mattangus]

I hacked together this. This will only work for cmake projects:

example

from conan.tools.cmake.cmakedeps.templates.target_configuration import TargetConfigurationTemplate
from conans.client.graph.graph import RECIPE_EDITABLE
import textwrap


@property
def context(self):
    deps_targets_names = self.get_deps_targets_names() if not self.require.build else []

    components_targets_names = self.get_declared_components_targets_names()
    components_names = [
        (components_target_name.replace("::", "_"), components_target_name)
        for components_target_name in components_targets_names
    ]

    is_win = self.conanfile.settings.get_safe("os") == "Windows"
    auto_link = self.cmakedeps.get_property(
        "cmake_set_interface_link_directories", self.conanfile
    )
    is_editable = (
        "TRUE"
        if self.conanfile._conanfile._conan_node.recipe == RECIPE_EDITABLE
        else "FALSE"
    )
    return {
        "pkg_name": self.pkg_name,
        "root_target_name": self.root_target_name,
        "config_suffix": self.config_suffix,
        "config": self.configuration.upper(),
        "deps_targets_names": ";".join(deps_targets_names),
        "components_names": components_names,
        "configuration": self.cmakedeps.configuration,
        "set_interface_link_directories": auto_link and is_win,
        "is_editable": is_editable,
    }


@property
def template(self):
    return textwrap.dedent(
        """\
    # Avoid multiple calls to find_package to append duplicated properties to the targets
    include_guard()

    {%- macro tvalue(pkg_name, comp_name, var, config_suffix) -%}
        {{'${'+pkg_name+'_'+comp_name+'_'+var+config_suffix+'}'}}
    {%- endmacro -%}

    ########### VARIABLES #######################################################################
    #############################################################################################
    set({{ pkg_name }}_FRAMEWORKS_FOUND{{ config_suffix }} "") # Will be filled later
    conan_find_apple_frameworks({{ pkg_name }}_FRAMEWORKS_FOUND{{ config_suffix }} "{{ '${' }}{{ pkg_name }}_FRAMEWORKS{{ config_suffix }}}" "{{ '${' }}{{ pkg_name }}_FRAMEWORK_DIRS{{ config_suffix }}}")

    set({{ pkg_name }}_LIBRARIES_TARGETS "") # Will be filled later

    set(IS_EDITABLE {{ is_editable }})
    ######## Create an interface target to contain all the dependencies (frameworks, system and conan deps)
    if(NOT TARGET {{ pkg_name+'_DEPS_TARGET'}})
        add_library({{ pkg_name+'_DEPS_TARGET'}} INTERFACE IMPORTED)
    endif()

    set_property(TARGET {{ pkg_name + '_DEPS_TARGET'}}
                    PROPERTY INTERFACE_LINK_LIBRARIES
                    $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_FRAMEWORKS_FOUND'+config_suffix+'}' }}>
                    $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_SYSTEM_LIBS'+config_suffix+'}' }}>
                    $<$<CONFIG:{{configuration}}>:{{ deps_targets_names }}>
                    APPEND)

    ####### Find the libraries declared in cpp_info.libs, create an IMPORTED target for each one and link the
    ####### {{pkg_name}}_DEPS_TARGET to all of them
    if (NOT ${IS_EDITABLE})
        conan_package_library_targets("{{ '${' }}{{ pkg_name }}_LIBS{{ config_suffix }}}"    # libraries
                                        "{{ '${' }}{{ pkg_name }}_LIB_DIRS{{ config_suffix }}}" # package_libdir
                                        "{{ '${' }}{{ pkg_name }}_BIN_DIRS{{ config_suffix }}}" # package_bindir
                                        "{{ '${' }}{{ pkg_name }}_LIBRARY_TYPE{{ config_suffix }}}"
                                        "{{ '${' }}{{ pkg_name }}_IS_HOST_WINDOWS{{ config_suffix }}}"
                                        {{ pkg_name + '_DEPS_TARGET'}}
                                        {{ pkg_name }}_LIBRARIES_TARGETS  # out_libraries_targets
                                        "{{ config_suffix }}"
                                        "{{ pkg_name }}"    # package_name
                                        "{{ '${' }}{{ pkg_name }}_NO_SONAME_MODE{{ config_suffix }}}")  # soname
    else()
        add_subdirectory({{ '${' }}{{ pkg_name }}_PACKAGE_FOLDER{{ config_suffix }}} {{ pkg_name }})
    endif()

    # FIXME: What is the result of this for multi-config? All configs adding themselves to path?
    set(CMAKE_MODULE_PATH {{ '${' }}{{ pkg_name }}_BUILD_DIRS{{ config_suffix }}} {{ '${' }}CMAKE_MODULE_PATH})
    {% if not components_names %}

    ########## GLOBAL TARGET PROPERTIES {{ configuration }} ########################################
        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_LINK_LIBRARIES
                        $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_OBJECTS'+config_suffix+'}' }}>
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_LIBRARIES_TARGETS}>
                        APPEND)

        if("{{ '${' }}{{ pkg_name }}_LIBS{{ config_suffix }}}" STREQUAL "")
            # If the package is not declaring any "cpp_info.libs" the package deps, system libs,
            # frameworks etc are not linked to the imported targets and we need to do it to the
            # global target
            set_property(TARGET {{root_target_name}}
                            PROPERTY INTERFACE_LINK_LIBRARIES
                            {{pkg_name}}_DEPS_TARGET
                            APPEND)
        endif()

        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_LINK_OPTIONS
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_LINKER_FLAGS{{config_suffix}}}> APPEND)
        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_INCLUDE_DIRECTORIES
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_INCLUDE_DIRS{{config_suffix}}}> APPEND)
        # Necessary to find LINK shared libraries in Linux
        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_LINK_DIRECTORIES
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_LIB_DIRS{{config_suffix}}}> APPEND)
        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_COMPILE_DEFINITIONS
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_COMPILE_DEFINITIONS{{config_suffix}}}> APPEND)
        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_COMPILE_OPTIONS
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_COMPILE_OPTIONS{{config_suffix}}}> APPEND)

        {%- if set_interface_link_directories %}

        # This is only used for '#pragma comment(lib, "foo")' (automatic link)
        set_property(TARGET {{root_target_name}}
                        PROPERTY INTERFACE_LINK_DIRECTORIES
                        $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_LIB_DIRS{{config_suffix}}}> APPEND)
        {%- endif %}


    {%- else %}

    ########## COMPONENTS TARGET PROPERTIES {{ configuration }} ########################################

        {%- for comp_variable_name, comp_target_name in components_names %}


        ########## COMPONENT {{ comp_target_name }} #############

            set({{ pkg_name }}_{{ comp_variable_name }}_FRAMEWORKS_FOUND{{ config_suffix }} "")
            conan_find_apple_frameworks({{ pkg_name }}_{{ comp_variable_name }}_FRAMEWORKS_FOUND{{ config_suffix }} "{{ '${'+pkg_name+'_'+comp_variable_name+'_FRAMEWORKS'+config_suffix+'}' }}" "{{ '${'+pkg_name+'_'+comp_variable_name+'_FRAMEWORK_DIRS'+config_suffix+'}' }}")

            set({{ pkg_name }}_{{ comp_variable_name }}_LIBRARIES_TARGETS "")

            ######## Create an interface target to contain all the dependencies (frameworks, system and conan deps)
            if(NOT TARGET {{ pkg_name + '_' + comp_variable_name + '_DEPS_TARGET'}})
                add_library({{ pkg_name + '_' + comp_variable_name + '_DEPS_TARGET'}} INTERFACE IMPORTED)
            endif()

            set_property(TARGET {{ pkg_name + '_' + comp_variable_name + '_DEPS_TARGET'}}
                            PROPERTY INTERFACE_LINK_LIBRARIES
                            $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_'+comp_variable_name+'_FRAMEWORKS_FOUND'+config_suffix+'}' }}>
                            $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_'+comp_variable_name+'_SYSTEM_LIBS'+config_suffix+'}' }}>
                            $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_'+comp_variable_name+'_DEPENDENCIES'+config_suffix+'}' }}>
                            APPEND)

            ####### Find the libraries declared in cpp_info.component["xxx"].libs,
            ####### create an IMPORTED target for each one and link the '{{pkg_name}}_{{comp_variable_name}}_DEPS_TARGET' to all of them
            conan_package_library_targets("{{ '${'+pkg_name+'_'+comp_variable_name+'_LIBS'+config_suffix+'}' }}"
                                    "{{ '${'+pkg_name+'_'+comp_variable_name+'_LIB_DIRS'+config_suffix+'}' }}"
                                    "{{ '${'+pkg_name+'_'+comp_variable_name+'_BIN_DIRS'+config_suffix+'}' }}" # package_bindir
                                    "{{ '${'+pkg_name+'_'+comp_variable_name+'_LIBRARY_TYPE'+config_suffix+'}' }}"
                                    "{{ '${'+pkg_name+'_'+comp_variable_name+'_IS_HOST_WINDOWS'+config_suffix+'}' }}"
                                    {{ pkg_name + '_' + comp_variable_name + '_DEPS_TARGET'}}
                                    {{ pkg_name }}_{{ comp_variable_name }}_LIBRARIES_TARGETS
                                    "{{ config_suffix }}"
                                    "{{ pkg_name }}_{{ comp_variable_name }}"
                                    "{{ '${'+pkg_name+'_'+comp_variable_name+'_NO_SONAME_MODE'+config_suffix+'}' }}")


            ########## TARGET PROPERTIES #####################################
            set_property(TARGET {{comp_target_name}}
                            PROPERTY INTERFACE_LINK_LIBRARIES
                            $<$<CONFIG:{{configuration}}>:{{ '${'+pkg_name+'_'+comp_variable_name+'_OBJECTS'+config_suffix+'}' }}>
                            $<$<CONFIG:{{configuration}}>:${{'{'}}{{pkg_name}}_{{comp_variable_name}}_LIBRARIES_TARGETS}>
                            APPEND)

            if("{{ '${' }}{{ pkg_name }}_{{comp_variable_name}}_LIBS{{ config_suffix }}}" STREQUAL "")
                # If the component is not declaring any "cpp_info.components['foo'].libs" the system, frameworks etc are not
                # linked to the imported targets and we need to do it to the global target
                set_property(TARGET {{comp_target_name}}
                                PROPERTY INTERFACE_LINK_LIBRARIES
                                {{pkg_name}}_{{comp_variable_name}}_DEPS_TARGET
                                APPEND)
            endif()

            set_property(TARGET {{ comp_target_name }} PROPERTY INTERFACE_LINK_OPTIONS
                            $<$<CONFIG:{{ configuration }}>:{{tvalue(pkg_name, comp_variable_name, 'LINKER_FLAGS', config_suffix)}}> APPEND)
            set_property(TARGET {{ comp_target_name }} PROPERTY INTERFACE_INCLUDE_DIRECTORIES
                            $<$<CONFIG:{{ configuration }}>:{{tvalue(pkg_name, comp_variable_name, 'INCLUDE_DIRS', config_suffix)}}> APPEND)
            set_property(TARGET {{comp_target_name }} PROPERTY INTERFACE_LINK_DIRECTORIES
                            $<$<CONFIG:{{ configuration }}>:{{tvalue(pkg_name, comp_variable_name, 'LIB_DIRS', config_suffix)}}> APPEND)
            set_property(TARGET {{ comp_target_name }} PROPERTY INTERFACE_COMPILE_DEFINITIONS
                            $<$<CONFIG:{{ configuration }}>:{{tvalue(pkg_name, comp_variable_name, 'COMPILE_DEFINITIONS', config_suffix)}}> APPEND)
            set_property(TARGET {{ comp_target_name }} PROPERTY INTERFACE_COMPILE_OPTIONS
                            $<$<CONFIG:{{ configuration }}>:{{tvalue(pkg_name, comp_variable_name, 'COMPILE_OPTIONS', config_suffix)}}> APPEND)

            {%- if set_interface_link_directories %}
            # This is only used for '#pragma comment(lib, "foo")' (automatic link)
            set_property(TARGET {{ comp_target_name }} PROPERTY INTERFACE_LINK_DIRECTORIES
                            $<$<CONFIG:{{ configuration }}>:{{tvalue(pkg_name, comp_variable_name, 'LIB_DIRS', config_suffix)}}> APPEND)

            {%- endif %}
        {%endfor %}


        ########## AGGREGATED GLOBAL TARGET WITH THE COMPONENTS #####################
        {%- for comp_variable_name, comp_target_name in components_names %}

        set_property(TARGET {{root_target_name}} PROPERTY INTERFACE_LINK_LIBRARIES {{ comp_target_name }} APPEND)

        {%- endfor %}


    {%- endif %}


    ########## For the modules (FindXXX)
    set({{ pkg_name }}_LIBRARIES{{ config_suffix }} {{root_target_name}})

    """
    )


TargetConfigurationTemplate.template = template
TargetConfigurationTemplate.context = context

The core change being

    ...
    set(IS_EDITABLE {{ is_editable }})
    ...
    if (NOT ${IS_EDITABLE})
        conan_package_library_targets("{{ '${' }}{{ pkg_name }}_LIBS{{ config_suffix }}}"    # libraries
                                        "{{ '${' }}{{ pkg_name }}_LIB_DIRS{{ config_suffix }}}" # package_libdir
                                        "{{ '${' }}{{ pkg_name }}_BIN_DIRS{{ config_suffix }}}" # package_bindir
                                        "{{ '${' }}{{ pkg_name }}_LIBRARY_TYPE{{ config_suffix }}}"
                                        "{{ '${' }}{{ pkg_name }}_IS_HOST_WINDOWS{{ config_suffix }}}"
                                        {{ pkg_name + '_DEPS_TARGET'}}
                                        {{ pkg_name }}_LIBRARIES_TARGETS  # out_libraries_targets
                                        "{{ config_suffix }}"
                                        "{{ pkg_name }}"    # package_name
                                        "{{ '${' }}{{ pkg_name }}_NO_SONAME_MODE{{ config_suffix }}}")  # soname
    else()
        add_subdirectory({{ '${' }}{{ pkg_name }}_PACKAGE_FOLDER{{ config_suffix }}} {{ pkg_name }})
    endif()
    ...

This is then quite seamless. When I change test_core_lib, it only recompiles the source files I changed.

[mattangus]

Not trying to spam you, but after converting one of my projects to conan I found another "issue" with not being able to use cmake for triggering the build process. If I build without any changes on my project with quite a few (transitive) dependencies it takes 7s to "build".

requirements

        self.requires("fmt/10.2.1")
        self.requires("rapidjson/cci.20230929")
        self.requires("eigen/3.4.0")
        self.requires("opencv/4.8.1") 
        self.requires("libpng/1.6.42", override=True)

$ time conan build . --build=editable
# 7.249 total
$ time cmake --preset conan-release
# 0.174 total

The good news is that it doesn't seem to scale much more when I add other dependencies. Going from ~30 to ~60 (including transitive), it only goes up to 7.524s.

[mattangus]

I think the answer is workspaces so I'll just wait for that to come to conan 2.

[memsharded]

If you think that is a viable route, would you accept a PR doing something like this?

I am definitely interested in getting these ideas incorporated into the workspace discussion.

I have been playing around with the workspace for 2.0, but I hadn't considered your approach of replacing conan_package_library_targets with add_subdirectory in CMakeDeps. I think this is an interesting idea worth exploring, I am creating at new ticket to centralize efforts of workspaces in #15992, lets follow up there. Please track the issue, we will layout the basics first, and when working on the CMake setup, we will continue with this, thanks for the offer!

[memsharded]

$ time conan build . --build=editable

The conan build actually builds the whole dependency graph, launches the build of the dependencies too with configure step, not only the build step, so it will be doing quite more than cmake, so the slower times are expected. Indeed one goal of workspaces is to reduce this time.