[feature] Conan generator to mimic submodules behavior

[sin-diesel]

What is your suggestion?

Hi! I've recently worked on a set of projects which, by the nature of their content, are submodule-based. One of the most severe issues that I've encontered with submodules that is really problematic and widespread is the duplication of repositories included multiple times in the submodules tree structure.
We've thought of starting to use conan there, because the conan cache is shared and, if we wrap all submodules in conan packages, we'll get rid of the problem of multiple duplication of the same code in packages.
Though currently conan supports that, the facilities it provided are not sufficient to comfortably work with source packages.

First of all, we lose the repository tree structure. Developers really like that and need to work with source packages in the source tree. instead of going to cache every time they need to make changes to the source code.

Second of all, sometimes we need to make sure that not every developer can download and install all submodules in the repository. Sometimes access to them should be limited and only people with direct access to git repository can get the source package with it source code.

The solution I propose is following:

1. We do not put source code into packages. Instead, we put git url and commit sha, which solves the first problem of managing access to repositories. Only people with direct access can clone the source code by url. Those packages are in conan cache, but repositories are not yet cloned, only packages with urls and shas are downloaded and installed in cache.
2. We write conan generator, which, in its turn, will traverse through installed packages and use url and sha to clone the git repository, but now in repository source tree instead. If access is denied, conan will continue cloning other repositories and will throw a warning.

Each repo can still be individually managed (making commits, creating conan packages, etc). This will solve the problem of source code duplication in submodules using conan cache, and will still provide transparent workflow for developers who are required to work with submoduled.

Have you read the CONTRIBUTING guide?

• I've read the CONTRIBUTING guide

[memsharded]

Hi @sin-diesel

Thanks for your feedback.

First of all, we lose the repository tree structure. Developers really like that and need to work with source packages in the source tree. instead of going to cache every time they need to make changes to the source code.

It is important to understand that package-based development is basically a different development paradigm than mono-repo development. Once package-based development is used, and different packages can be developed and released independently under its own version, some features that could be used in a mono repo approach are lost.

Conan packages have fully independent builds to the point that different packages can have completely different build systems and a package generated with any build system can be consumed by any package with any other build system.

This introduces a high decoupling between packages. Even if they use the same build system, as CMake, the features used in both cases are different, package-based development uses mostly find_package() with IMPORTED targets, and mono-repo development uses add_subdirectory() with project regular targets. That means that it is basically impossible to have the same development experience than a mono repo and have also the same advantages as package-based with the same ergonomics.

I think what you are looking for might is the editables packages (have a look to https://docs.conan.io/2/tutorial/developing_packages/editable_packages.html) a way to work simultaneously on multiple packages in user space (but not using the same mono-project, but multiple projects), and the planned workspace feature (we plan to resume this in the following months). Even the workspace feature is not yet available, doing some scripting or even manually as seen in https://www.youtube.com/live/VzUJQw89U7o?si=ryTSwLCUbrLBqYZK&t=2518, it is possible to create a meta-project that gathers all the different projects and connects them for developer convenience.

Second of all, sometimes we need to make sure that not every developer can download and install all submodules in the repository. Sometimes access to them should be limited and only people with direct access to git repository can get the source package with it source code.

This feature, using Conan can certainly help, because it is easier to control the package binaries so not all developers have access to the source code but to the binaries only.

We do not put source code into packages. Instead, we put git url and commit sha, which solves the first problem of managing access to repositories. Only people with direct access can clone the source code by url. Those packages are in conan cache, but repositories are not yet cloned, only packages with urls and shas are downloaded and installed in cache.
We write conan generator, which, in its turn, will traverse through installed packages and use url and sha to clone the git repository, but now in repository source tree instead. If access is denied, conan will continue cloning other repositories and will throw a warning.

The features you suggest will be part of the workspace feature in #15992, mostly being able to clone any arbitrary dependency using the scm feature as described in https://docs.conan.io/2/examples/tools/scm/git/capture_scm/git_capture_scm.html, to be added automatically to the current workspace. This is even drafted in one of my branches. The most challenging part is to put together the different packages under the same build-system project, and there is no good solution for it. We will propose some tools and utilities in the workspace feature, but it is likely that users will still have to define their own meta-project logic.

As I think this would be mostly responded by editables and workspaces, I'd suggest to close this one as duplicated and track #15992

[memsharded]

Hi @sin-diesel

Any further feedback here? Can we close this ticket and follow up on the workspace feature ticket #15992?

[sin-diesel]

Hi @memsharded
Sure, thanks!