Best way to set up conan for multiple subprojects

[fittyCent]

Situation

• single C++ repo containing 20 projects
• Builds in MSVS for windows, uses cmake for Linux. Same code base builds for both platforms.
  • There's an msvs solution file connecting all projects in the repo together
• Trying to find best tools and workflow

What I have so far for Windows

• installed conan extension for msvs
• I have a conanfile.txt for a couple of projects (not all projects need them) and they're located in the root folder of each project (not the root folder of the repo).
• There's a "conan install for all projects in solution" button but that does nothing somehow.
• Question1: Is there an easier way for someone to install conan deps without having to right click on each project and select "Conan install for this project"?
• Conan extension's default install location is $(OutputDir)\.conan. So when I "conan install" a project through the tool, there are important conan*.props that the project would import. These props are now in the output folder (e.g. x64/Debug) that's ignored by git. If I pushed the repo in this state, the next guy who clones it won't be able to open the solution because the projects now reference the props but they're not present.
  • Question2 I've changed the install location to $(SolutionDir)\.conan. Is this the common way to do it?
  • Question3 Is there a way to default the conan msvs extension (maybe through settings files) so a new machine wouldn't go back to installing to $(OutputDir)\.conan and screwing up the repo?

I could be doing this all wrong so any best practices tips will be greatly appreciated.

• I've read the CONTRIBUTING guide.

[memsharded]

Hi @fittyCent

First, some questions to understand your case:

• Do you have a project with 20 subprojects, but do you want to create packages for every subproject?
• Do you want to just install third party dependencies for your project, and different subprojects might use different ones? For example, subproject1 will depend on zlib and bzip2, and subproject2 will depend on boost and zlib, etc
• Should the third party dependencies versions be consistent or they need to be different? That is, all subprojects will use the same zlib version, or for some reason there is a subproject wanting to use zlib/1.2.8 and another wanting to use zlib/1.2.11?

If the case is using third party dependencies my recommendation would be:

• Have 1 single conanfile with all the dependencies the project need
• Use a conanfile.py, because you can define there the generate() method to customize behavior of generators if needed, and you can define layout() to better specify where do you want to put the files (self.folders.generators)
• Use modern generators, CMakeDeps and CMakeToolchain and MSBuildDeps
• Just do 1 conan install, better drop the VS extension, as it is not maintained at this moment and its functionality is limited. The flow will be the same for both systems, CMake and VS (the flow is git clone ... + conan install ... + cmake (or open .sln))
• Each subproject can add their specific .props or use the specific find_package() for the dependencies they use. Transitivity is automatically managed, only need to specify the direct dependencies.

[fittyCent]

Thanks @memsharded. That helped A TON!

I am just consuming third party dependencies and your workflow makes a whole lot of sense. I'm still working on ironing out all the kinks and I'll close this discussion as soon as I confirm this workflow doesn't hit roadblocks (if you don't mind).

Seems like most of the docs I read indicated that conanfile.py is used only when creating distributables. This file turned out to be really handy for consuming as well.

[memsharded]

Thanks @memsharded. That helped A TON!

Happy to hear that! :) :)
We are here to help. Don't hesitate to raise further questions you might have

Seems like most of the docs I read indicated that conanfile.py is used only when creating distributables. This file turned out to be really handy for consuming as well.

Yes, we are aware of this. This is the reason that for the new 2.0 docs (https://docs.conan.io/en/2.0-alpha/tutorial/consuming_packages.html), we are giving way more relevance to using conanfile.py as consumer, introducing it in the main flow of the main getting started tutorial.

[fittyCent]

hey @memsharded, thanks for your response. I got pulled off to do something else and I'm back at this, sorry.

I'm stuck somewhere and I've been trying to get through this for some time now. Here's my setup so far:

• repo1
  • conanfile.py
    • require dependency called redis-plus-plus
    • Windows generator: MSBuildDeps
    • Linux generators: CMakeDeps and CMakeToolchain
    • layout() points all generated files to repo1/conan
      • This generates files for redis-plus-plus and its dep called hiredis
  • project1
    • builds static lib
    • CMakeLists.txt file
      • include(${CMAKE_SOURCE_DIR}/conan/redis++-config.cmake)
      • adds ${redis++_INCLUDE_DIR} to include list
      • adds ${redis++_LIBRARIES} to target libraries
  • project1a
    • builds executable
    • links to static lib created by project1
    • CMakeLists.txt file
      • include(${CMAKE_SOURCE_DIR}/conan/redis++-config.cmake)
      • add ${redis++_INCLUDE_DIR} to include list
      • add ${redis++_LIBRARIES} to target libraries
    • successfully built.
• repo2
  • project2
    • links to static lib created by project1
    • I'm getting this error

    CMake Error at project2/CMakeLists.txt:24 (add_executable):
      Target "project2" links to target "hiredis::hiredis" but
      the target was not found.  Perhaps a find_package() call is missing for an
      IMPORTED target, or an ALIAS target is missing?
    

    • tried adding include(<path to repo1>/conan/redis++-config.cmake) to CMakeLists.txt of project2 and still doesn't work. Tried to add redis++ include and libs to project2, and then tried doing all this with hiredis instead, all didn't work.

Any idea how I'd make this work? Thanks in advance!!

[memsharded]

Hi @fittyCent

I think the main conceptual thing is that you are trying to define a dependency from project2->project1 without actually modelling such dependency. When I was taking above about 1 conanfile to serve multiple subprojects, I was assuming there were actual subprojects, in which you can leverage CMake to target_link_libraries() the other targets from other add_subdirectory().

But when I see your links to static lib created by project1 it seems that you are providing the path to the lib somehow directly, but not leveraging CMake.

Also you should probably not be doing include(<path to repo1>/conan/redis++-config.cmake), those files are not intended for inclusion, but for find_package() behavior. Also, just adds ${redis++_INCLUDE_DIR} to include list doesn't work, because you are basically dropping all the transitivity information in the targets, you need to do target_link_libraries(my_target PUBLIC redis::redis) or something like that.

The thing is that CMake doesn't know how to connect independent projects, and something is necessary, there are different alternatives:

• To put all repos under an umbrella folder, that contains a root CMakeLists.txt that add_subdirectory() each one of the repos. Then use target_link_libraries(.... mydep::mydep) with real targets, as they keep information from transitive dependencies.
• To use some CMake mechanism like ExternalProjectAdd or similar.
• To use some Conan utilities like editable packages to model the dependency from project2->project1

I would probably try the first one first. I would try to avoid the second, and if I needed more power, like making project1 an actual Conan package (this might make sense, as it allows to automate its creation, different binaries, etc), then the last one using editables might be a solution.

Please let me know if this clarifies it a little bit.
Another possibility is to set up a toy github repo, with simplified "Hello world" versions of the libraries and projects, and a script that automates the steps. That way we can collaborate towards a fully working example.

[fittyCent]

Hi @memsharded

Sorry for the late post. I wanted to get something working with my weird set up before I responded. Here's what I did:

• repo1
  • conanfile.py
    • require dependency called redis-plus-plus
    • Windows generator: MSBuildDeps
    • Linux generators: CMakeDeps and CMakeToolchain
    • layout() points all generated files to repo1/conan
      • This generates files for redis-plus-plus and its dep called hiredis
  • repo1/CMakeLists.txt
    • include(repo1/conan/conan_toolchain.cmake)
    • add_subdirectory(project1)
  • project1
    • builds static lib
    • CMakeLists.txt file
      • find_package(redis++ REQUIRED)
      • add ${redis++_INCLUDE_DIR} to include list
      • add ${redis++_LIBRARIES} to target libraries
      • install(TARGETS project1 EXPORT project1Targets DESTINATION ${CMAKE_BINARY_DIR}/libs)
      • install(FILES cmake/project1Config.cmake DESTINATION ${CMAKE_BINARY_DIR}/libs)
      • install(EXPORT project1Targets FILE project1Targets.cmake NAMESPACE project1:: DESTINATION ${CMAKE_BINARY_DIR}/libs)
      • export(TARGETS project1 NAMESPACE project1:: FILE project1Targets.cmake)
  • project1a
    • builds executable
    • links to static lib created by project1
    • CMakeLists.txt file
      • find_package(redis++ REQUIRED)
      • add ${redis++_INCLUDE_DIR} to include list
      • add ${redis++_LIBRARIES} to target libraries
    • successfully built.
• repo2
  • project2
    • links to static lib created by project1
      • set(project1_DIR repo1/build/libs) # repo1/build/libs is ${CMAKE_BINARY_DIR}/libs from above
      • find_package(project1 REQUIRED)
      • target_link_libraries(project2 project1::project1)
    • successfully built

Thanks for your help and pointers on what I was doing wrong!

[memsharded]

Hi @fittyCent

Thanks for following up and reporting your setup.
Happy to know that you managed to get a setup that works for you, and the pointers helped.

Just one small recommendation:

• The include(repo1/conan/conan_toolchain.cmake) is probably not good practice, the toolchains are intended to be passed with cmake ... -DCMAKE_TOOLCHAIN_FILE=xxxx and not included

Also, at some point (not necessarily now, but later) you might want to try the editable features, as a way to connect different packages in user spaces. That means leveraging some of the Conan functionality and make "project1" a Conan package, instead of some of the custom CMake stuff you did. But if you are comfortable with your setup so far, that is good for now. Cheers!

[VladyslavOdobesku-TomTom]

@memsharded Hi, I'm currently googling things and maybe you know something about it as I barely see any info about that.
https://www.youtube.com/watch?v=VzUJQw89U7o&t=2437s

Small recap. In this video it is presented how nice conan editable works especially in case of visual studio when you can simply generate project for A then generate project for B that depends on A (A is placed into conan editable) and then you can simply add A project into B and it nicely works together.

Question is whether it is possible to do something like that on linux/macOS? Is it possible to do that with CLion IDE?

[memsharded]

Question is whether it is possible to do something like that on linux/macOS? Is it possible to do that with CLion IDE?

I don't think so, because that depends on the IDE. CLion being cmake-based, it is not possible to plug-in subprojects like in VS. Linux/MacOS, depends on the build system mostly (the underlying build-system, not CMake), so in general it is not possible.

For those environments the recomendation would be:

• Do not use workspace feature, as it is not maintained in 1.X and has been removed in 2.0
• Create a small setup script that calling conan editable and some of your project logic can create a meta-project, like a parent CMakeLists.txt that add_subdirectory the subprojects
• Track this ticket: Is workspace feature supported in Conan 2.0? #12466, and give feedback there. We will resume working on workspaces for Conan 2.0 in the Conan 2.X roadmap