Conan 2 workspaces

[memsharded]

This ticket purpose is to gather and centralize discussion of all the other opened tickets, now that we are starting to resume work on workspaces for Conan 2.

• The idea of using CMakeDeps in [question] Can an editable package be build using only cmake commands #15880 (comment) by @mattangus needs to be explored
• Adding the capability of having local configuration (custom commands) that compose with the global cache custom commands [feature] Folder specific custom commands #16023
• Adding the capability of having local config (global.conf) in a project, and compose it with the global one: [feature] On-the-fly configuration for 2.0 #6761
• Being able to have "light" workspaces which are mostly just a list of editables: Lightweight workspaces aka Temporary editables #5566
• CMake necessary changes to make the old 1.X workspace work: [workspaces] update documentation #7206
• [question] Shared code and workspaces #5939 seems it would have been solved with new layout() definition
• The issue [workspaces] Workspaces should be Super Builds #5762 contains a large discussion, the globalness of editables need to be addressed (probably the first issue a workspace 2.X will solve)
• Issue [workspaces] Decoupled Workspaces #6844 suggests more decoupled workspaces
• Ticket [feature] Extend .conanrc for profiles information #15209 suggest to be able to have profiles locally defined by .conanrc. If considered, this would be a workspace feature.

[aander80]

I am very keen on seeing workspaces being introduced!

Currently, I can think of two features that I would like to have.

Single build command for building multiple packages in a workspace

I have created a fork of your own fork and implemented a very hacky solution of this: https://github.com/aander80/memsharded-conan/tree/feature/workspace. I believe you should be able to clone my fork, checkout feature/workspace, cd into the workspace folder and run ./demo.sh (I ran it in WSL, have not tested it in Windows).

The idea is to have a command like conan workspace build <targets> (conan ws:build <targets> in my fork) and have conan build all the targets and editable dependencies in the workspace given the build order graph (note: I was not sure how to get this correctly using the API so my solution is probably not fully correct, but it seems to work for my demo). If <targets> is empty, the entire workspace is built. Of course, this would require efficient caching in each build.

On-the-fly editables and overriding dependencies

I have been looking into Cargo workspaces for Rust. There you have the wonderful feature of setting dependencies for the workspace, and in each package/crate re-use the workspace dependencies. Example here:

• Workspace: https://github.com/astral-sh/uv/blob/main/Cargo.toml#L19
• Package/crate: https://github.com/astral-sh/uv/blob/main/crates/uv-cli/Cargo.toml#L16

I imagine something similar would be very powerful for conan, where you can select to override dependencies in a workspace. When you are working with large projects with many commits being merged every day and with dynamic versioning where each commit gets a unique version, managing static editables will be tough to manage (especially since just creating a new commit during local development would force you to change your editables).

To solve this, it would be very useful if you have multiple repos, add them to a workspace, and the dependencies would be automatically injected in the dependency graph as editables, without having to configure them statically as editables in the conan cache.

In my example, let's assume I have two repos: say and hello. Here, I likely always want to make sure I use my local copy of say, and not the latest version in the cache (which might override my local copy/repo). I have currently thought of these different behaviors:

• You always inject/override dependencies in the graph. If hello points to say/[~1.0] but I have say/1.1.0 in my local repo, Conan would still select my local say/1.1.0 version.
• Alternatively, this could be an option/additional parameter that you set, similar to include_prerelease. I am not creative when it comes to names, but it could be something along the lines of allow_workspace_override. This could also be a core configuration that you can set to apply to all dependencies, similar to core.version_ranges:resolve_prereleases.
• It could also be that you must conform to the version ranges in the version ranges. However, Conan could prioritize the workspace versions ahead of the cache, so that if I have say/1.1.0 in my local repo but there is also say/1.1.1 in my cache, the dependency say/[~1.1] would still resolve to my workspace say/1.1.0. (I guess this is identical to the previous item in this list, with the allow_workspace_override feature turned off)

I understand that this could be very tricky to solve since you might have multiple versions and contexts of packages; some might require say/1.0, another say/2.0 etc., some might want to use say as a require, other as a tool-require and so on. I imagine this could be managed by a combination of the second and third items in the behavior list above, but I am probably missing loads here.

For me I think a key feature for simplicity is to have on-the-fly editables that are only injected in the workspace/for the current command.

I hope I managed to make myself understood in my ramblings here! 😄

[memsharded]

Hi @aander80

Thanks for your feedback.

Regarding your first point:

The idea is to have a command like conan workspace build (conan ws:build in my fork) and have conan build all the targets and editable dependencies in the workspace given the build order graph (note: I was not sure how to get this correctly using the API so my solution is probably not fully correct, but it seems to work for my demo). If is empty, the entire workspace is built. Of course, this would require efficient caching in each build.

This can be done today with the conan install <path-to-root> --build=editable. This will call the build() of every package in editable mode in the current dependency graph of <path-to-root> in the right order.

This will be most likely improved with a conan workspace build or similar, but that command will very likely just be a --build=editable under the hood, leveraging some definition in the workspace of the "path-to-root".

About your second point, I think it is very aligned with our current vision, and I have already started to work on it, I have some working code in a branch. Keep tuned, I will try to share when possible to start gathering feedback, I will share it in this thread.

[aander80]

Thanks for your reply! I am happy to hear about your vision, looking forward to seeing this implemented some day! 😄 It is also great to hear building multiple editables is possible already today, that should make multirepo development easier already now, although it might require a few extra commands!

[memsharded]

Hi everyone,

This is finally happening, we have the first PR for next Conan 2.10: #17272

Feedback is very welcome, if you can run from my source branch, new conan workspace commands, and if you have any issues or questions, please open new tickets, one for each of them. Having early feedback, even before the Conan 2.10 release can be helpful, otherwise, it will be released in 2.10 for dev-testing only, not for production, but looking forward your feedback in any case. Thanks!

[memsharded]

#17272 has been merged, it will be in next Conan 2.10.

It needs env-var defined CONAN_WORKSPACE_ENABLE=will_break_next, as for this iteration it is intended for dev-testing, not for production. Really looking forward to hearing your feedback!

[mattangus]

I've just done some poking around the new command and I have a few initial thoughts

1. It would be good to have a conan workspace new command similar to conan new ... in order to generate the boilerplate workspace
   • Ideally with a few templates, e.g. monorepo where it searches each subfolder for a conanfile.py, basic where it just has an empty list of folders
2. It might be useful to have a way to clean the editable build folders. This is sometimes required when a cached variable isn't set properly or is changed. Going into each individual editable package and removing the build folder/cache is tedious.
3. conan workspace open might need a more clear description in the help message, it's not clear what it's for
4. This might be difficult but ideally once workspace is set up the user could just use cmake to rebuild the editable packages
   • i.e. app depends on project1, I make a change in project1 and run cmake --build --preset conan-release in the app folder, the change in project1 is detected and recompiled.
   • The main thought behind this is that with more complicated projects the dependencies can take a while to enumerate so running conan build ... each time you want to compile could increase incremental build times by a lot.
5. I'm not sure if it's my misunderstanding or something missing in workspaces but IDE integration seems like it will be difficult with the current implementation. e.g. clangd expects a single compilation database. But a monorepo with a workspace has a compilation database for each editable package.

[mattangus]

I think I've been able to come up with a reasonable solution for local development for point 5 (and probably point 4) above. You can add each project in the mono repo using fetch content with OVERRIDE_FIND_PACKAGE on, then you don't need to change your cmake code for all 3 cases: editable mode, normal conan build, and mono cmake build. You just have to make sure that the targets defined in your conanfiles match the targets defined by your cmake (which is probably best practice anyway).

Here is some cmake that got the job done in my little test repo:

cmake_minimum_required(VERSION 3.15)
project(monorepo CXX)

include(FetchContent)

function(add_project PROJECT_PATH_AND_NAME)
    FetchContent_Declare(
        ${PROJECT_PATH_AND_NAME}
        SOURCE_DIR ${CMAKE_CURRENT_LIST_DIR}/${PROJECT_PATH_AND_NAME}
        SYSTEM
        OVERRIDE_FIND_PACKAGE
    )
    FetchContent_MakeAvailable(${PROJECT_PATH_AND_NAME})
endfunction()

add_project(project1)
add_project(project2)
add_project(app)

You have to set the output directory for each sub project as well like so:

set_target_properties(project2
    PROPERTIES
    ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
    RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}"
)

Ideally they would be set to the following, but I can't figure out how to get conan to play nice with this.

set_target_properties(project2
    PROPERTIES
    ARCHIVE_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib"
    LIBRARY_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/lib"
    RUNTIME_OUTPUT_DIRECTORY "${CMAKE_BINARY_DIR}/bin"
)

[memsharded]

Thanks very much for the feedback @mattangus.
I am adding a conan new workspace template to the existing conan new command in #17538, if you want to check it. I have started by adding some libs and one exe, and a static workspace, not a dynamic one. Not sure if the dynamic one would be too much, but lets follow up in the PR #17538, thanks!

I'll follow up on the other items when possible, but looking good, so thanks very much!