Seeking Recommendations: monorepo that is distributed as individual projects

[phillipjohnston]

I'd like to have a way to organize my build system to support development as a monorepo, while still allowing for distribution of pieces as individual projects in a way that I can automate. The problem I'm facing now is that there is too high of a development overhead when making and testing changes that reach across different modules. This is achievable from an infrastructure perspective, but I can't figure out a way to make the Meson build work. The subproject/project() design makes it seem like I will need a different build system for this case. I'm hoping others have input on patterns that might work to keep Meson in play.

What I am not looking for is a debate about whether or not a monorepo is the right choice, why splitting a monorepo into distinct repositories for distribution will never work, or how Meson's subproject philosophy is amazing and I am stupid for not understanding that. I will not engage with any discussions along these lines.

Other Notes:

• The easiest thing that would support this is being able to check at the start of the Meson file whether or not I am a subproject... but nothing can proceed the project() call, so this is not an option.
• My first thought was to make the main source tree the subprojects directory, but my quick tests have me running into sandbox violations left and right - the subprojects folder can't be treated the same way that a normal folder can
• The sandboxing makes it seem like this directory structure MUST be flat (as well as the wrap error "ERROR: Directory key must be a name and not a path", which prevents something like directory = platforms/heres-a-platforms being used for organizational purposes). This is not ideal, I want to organize the modules with a sensible structure (e.g., drivers/, platforms/, libraries/). Is there a way to get around this?

Thankful for any suggestions the community might have!

[ilg-ul]

I am also struggling to find a way to use meson in my modular projects. Since my projects use multiple folders for dependencies, subproject() is not an option. (For things to work, subproject() should have a variant that accepts absolute paths to any location).

Without a better idea, I used subdir() to include the sub-projects into the top application, but I don't feel very comfortable with this solution.

My projects also are multi-platform, and I try to adjust various parts of the build programatically, based on the platform_name input option, but this also presents some challenges.

Being stuck with subdir(), the biggest difficulty I faced was to include the top library definitions in the tests, which are located in the tests folder, so one step down into the folders hierarchy, since subdir() rejects any input that is not a child.

The hack around this was to create a link in the tests folder, pointing to the parent (..); this seems somehow functional, but created a lot of other problems with the paths in the build folder.

So things are not that merry as I thought, and I'm also interested in finding a better solution to use meson in my projects.

[eli-schwartz]

The easiest thing that would support this is being able to check at the start of the Meson file whether or not I am a subproject... but nothing can proceed the project() call, so this is not an option.

It's not clear to me what you want to do with this information, surely a project is a project no matter what?

Other than that, I think the basic issue here would be solved by something I've wanted for a while to do, but never actually investigated doing. The basic idea would look something like this:

The top level meson.build isn't its own project and doesn't build anything. Basically what a monorepo of individual projects is, right? So, instead of using project() and shoving everything off into a flat list in subprojects/ you would use a special mode, metaproject(), which restricts the creation of targets and is dedicated to adding subproject()s. In exchange, you get to specify the path to the subproject in an unrestricted manner (or well, we can say "no nesting").
I think this strikes the right balance between the desires of monorepo users and the desires of people who want the organizational purity of the current subproject design.

Now the problem is just finding time to implement all this. :D

[ilg-ul]

... metaproject() ...

If you are considering adding some new concepts, perhaps you can also think on a solution to also cover the modular projects, with multiple source libraries as dependencies.

[phillipjohnston]

Hi Eli,

(Moved this to the thread for the response - first time using discussions)

The easiest thing that would support this is being able to check at the start of the Meson file whether or not I am a subproject... but nothing can proceed the project() call, so this is not an option.
It's not clear to me what you want to do with this information, surely a project is a project no matter what?

You're right, is_subproject isn't actually what I want and this wasn't the right way of expressing what I want. Ideally, I would be able to detect the context ("I am a top level project, I am a subproject, I am a subdir call") and adapt accordingly. If I knew I was a subdir call, I would filter out the project statement and some other functionality that would not be necessary in monorepo mode, saving time in configuration and other places.

The top level meson.build isn't its own project and doesn't build anything. Basically what a monorepo of individual projects is, right? So, instead of using project() and shoving everything off into a flat list in subprojects/ you would use a special mode, metaproject(), which restricts the creation of targets and is dedicated to adding subproject()s. In exchange, you get to specify the path to the subproject in an unrestricted manner (or well, we can say "no nesting").
I think this strikes the right balance between the desires of monorepo users and the desires of people who want the organizational purity of the current subproject design.

I do think this gets me closer to my goal.

Some questions to explore the idea further:

• If everything is a subproject, I will incur a configure time hit that reproduces all of the functionality that could instead be encapsulated in the main project (checking and build flags, finding tools, etc.). Would the metaproject still let me specify tools and flags in a way that allows me to alleviate my per-project configuration burden? (noting I would need to handle the subproject side myself with meson.is_subproject())
• Is there a good reason that no targets should be declared? Or can we refine that restriction?
  • For example, I would like to generate static analysis reports, automated code formatting, and perform other tooling operations at a repository-wide level, instead of having to invoke these targets for each individual subproject.

[phillipjohnston]

Hi Eli,

If you ping me whenever you do get time to work on this effort, I would be happy to test it out for you.

[phillipjohnston]

Thinking about folders, I'm also wondering if I can structure my build rules in a way that gets me what I want re: context-awareness.

A project's top-level meson.build file can contain the project() and standalone infrastructure. Then I could have a folder, maybe build/, which contains a meson.build file that describes targets. The top-level project will include build/meson.build, and the project's local meson.build will too.

The gotcha, of course, is that you can't do this if you need to call subdir() because you'd need a .. in the path (e.g., if you want to go into ../src and ../test)

[phillipjohnston]

Based on other discussions, this seems like the primary path to explore - find a build structure that can allow you to leave out the top-level build rules and project() call while still picking up the other rules related to sources..

[phillipjohnston]

I have a proof of concept that works using this approach. Instead of a build/ folder, I just push all logic related to the library components itself into src/, test/, etc. The top level meson.build file subdirs these folders, and all it contains otherwise is the project call, subproject requirements, and any reusable build machinery.

The monorepo build files bypass the top-level folder and reach into src/ and test/. In this case, the monorepo setup supplies the same project-level build machinery, or some things are left out because they are not needed in the specific case.

Individual projects within the monorepo can still be built in the standard way with the local project folder. What I haven't handled yet is the fact that it will then use the wrap system to download the files once again - would be nice if I could leverage the existing code.

[jpakkane]

It is difficult to give exact advise for things like this without further details on what you are actually developing and how, exactly, your things are currently factored in.

allowing for distribution of pieces as individual projects in a way that I can automate.

Are these project the kind of things that would be fully standalone and could be used on their own or do they only work together?

How do you want to split out those projects? Do you want to have projects drivers, libraries etch each of which would have separate directories for each platform or a platforma, platformb etc each of which would have a driver, library etc subdir? Or something else?

[phillipjohnston]

Hi Jussi,

It is difficult to give exact advise for things like this without further details on what you are actually developing and how, exactly, your things are currently factored in.

Consider a tree organized along these lines, each having multiple subfolders. The applications folder defines binary targets, the others define library targets (and maybe applications for unit tests). Multiple applications are supported in the same tree, for different hardware variations (handled with different hardware_platforms / processors and toolchain files), while all other code is kept "common" and doesn't change implementation with the host architecture.

src
├── applications
├── core
├── drivers
├── hardware_platforms
├── meson.build
├── os
├── platforms
├── processors
├── subsystems
├── test (centralized test infrastructure, sub-projects have their own tests)
└── utilities

Just to note, as well, that all of this was built once upon a time in the monorepo setup (singular project()), and it currently is built with the subproject setup where everything is "blown apart". The subproject setup is ideal for consumption, because you can limit yourself to the dependencies you need - no reason to have to deal with nRF52 code when you're working on an STM32, for example.

The maintenance problem is that it is easier to change, refactor, and test in one place, because it's easy to miss a necessary dependency update that isn't used in the current repository I'm working in. While I will get eventual CI build failure alerts because I have an interface mismatch, I'd still rather get it right the first time.

Are these project the kind of things that would be fully standalone and could be used on their own or do they only work together?

It's a mix. For example, drivers can be used standalone, utilities can be used standalone, subsystems can be used standalone.

Other pieces have dependencies within the system. For example, a "hardware platform" would pull in a processor dependency and drivers for whatever is on the board.

How do you want to split out those projects? Do you want to have projects drivers, libraries etch each of which would have separate directories for each platform or a platforma, platformb etc each of which would have a driver, library etc subdir? Or something else?

Something like "core" would be released standalone. For all other items (applications, drivers, platforms, etc.), each subdirectory within the "top-level" type directory would represent a unit of code that would then be sync'd to the standalone project repository.

[jpakkane]

The maintenance problem is that it is easier to change, refactor, and test in one place, because it's easy to miss a necessary dependency update that isn't used in the current repository I'm working in.

How are the repositories set up? Which things are in their own and which are not? Note that just because you use subprojects it does not mean that you can't have a monorepo for all your stuff. There are people who do that, including having release tarballs that can be built separately.

[phillipjohnston]

Everything is in its own repository.

Note that just because you use subprojects it does not mean that you can't have a monorepo for all your stuff.

But if I don't want a flat directory structure with 50+ folders and no organization, what is the solution? I want that organized into the desired tree, which I cannot figure out how to do with the subprojects restrictions.

[phillipjohnston]

I am a human being and I need some order :)

(If it matters - I don't care whether wrap files are at the top level. I'm not worried about supporting different versions for different subprojects. I don't have circular dependencies. I'm primarily concerned about the directory structure).

[jpakkane]

But if I don't want a flat directory structure with 50+ folders and no organization, what is the solution?

Based on your earlier description it seemed like there would be on the order of 10 subprojects. Where do these 50 come from? If you have that many, could it be possible that it has been split into to fine grained subprojects? It is perfectly fine to have multiple libraries and executables in a single subproject if that makes logical sense.

Even if there are 50 independent subprojects then whether having them in the same subdirectory or not is still a matter of taste. Both approaches have their advantages. For the flat case it is very simple to know what dependencies are available and where they can be found: you just look in one directory and even further that directory is the same for all Meson projects. This makes it easy to get to know new code bases. You know exactly how they are organized and where everything is. The "free form" style gives you more flexibility to set things up the way you want. The downside is that questions like "where is functionality X" or even "does functionality X exist anywhere in my tree" become harder to answer. Unless you remember everything about your project, you can't answer that without a whole lot of grepping.

This is even more important when using external dependencies. If you ever need auditability (and most big commercial projects do) then knowing where all external deps must be simplifies things.

I don't care whether wrap files are at the top level. I'm not worried about supporting different versions for different subprojects. I don't have circular dependencies

The mismatch here is that Meson absolutely has to care about these. Being a general purpose build system it can't just throw its hands up in the air and say "don't do that or it breaks and you get to keep both pieces". We have to provide a system that is robust against problems like these because people are going to do those things and they are going to face a ton of issues because of it.

[phillipjohnston]

I have deleted my previous responses because they just expressed my frustration at the way subproject conversations always seem to go.

The message I am receiving from you is that Meson subprojects have no suitable usage pattern that will support this need. If this is an incorrect understanding, I would certainly appreciate a pointer to any examples of monorepos of subprojects that are of a similar or greater level of size.

Otherwise, I will continue to explore ways to workaround subproject for my monorepo goal, likely relying on build structuring so I can exclude project()-level stuff by including a different directory. This seems achievable, but it is also disappointing because it means I cannot actually exercise the subproject system within my monorepo, and this is how it is primarily used in the field.

[phillipjohnston]

On this point, among others, I do think we crossed wires:

The mismatch here is that Meson absolutely has to care about these. Being a general purpose build system it can't just throw its hands up in the air and say "don't do that or it breaks and you get to keep both pieces". We have to provide a system that is robust against problems like these because people are going to do those things and they are going to face a ton of issues because of it.

I'm saying wrap files at the top level is different than not having my source files organized. I can live with a list of wrap files at ht etop level.

I do not have circular dependencies, and I don't have different versions of different libraries, and I don't have nested subprojects. So if Meson cares about that and so do I, and I satisfy those cares, what did I get wrong?

[ilg-ul]

Perhaps I'm missing the point, but, apart from maintenance issues, from the point of view of the build process in general and of the meson build process in particular, what is the difference between a mono-repo project with lots of sub-folders, and a modular, multi-repo project, where the dependencies are somehow brought into the development tree (for example by using Git submodules or by relying on a package manager to create copies or links to individual source libraries)?

[phillipjohnston]

From a meson perspective, you cannot support both subproject usage and subdir() usage with the same build files (e.g., where you might pull it into your source tree via submodule) - the former requires project() to be present, while the latter cannot have it present.

(Hope I didn't misunderstand the question)

[ilg-ul]

Sorry if the question was confusing, but I think that your question on how to organise builds applies equally to mono-repo projects where the content of individual folders might also be consumed separately as source library dependencies by other projects and to modular multi-repo projects, which are obviously intended to be consumed as source library dependencies.

And your previous answer also highlights my own dilema, should the source library dependencies be consumed as separate entities, or as regular folders of the same project?

Personally I could not use meson subproject() because my modular projects do not meet the strict requirement to have a single folder with all dependencies, so I have no experience at all with using subproject() and I had to resort to subdir(), but from a theoretical point of view, I think that a safe build system should be able to sandbox processing these dependencies, otherwise there is a risk of overriding definitions when including all dependencies in the same context.

If meson's answer to this is subproject(), fine, but it should be made more flexible to allow dependencies to be included from any locations in the filesystem. If subproject() was intended for other use cases and it is not reasonable to change it, then a new mechanism must be considered, but it definitely should allow to refer to source libraries located anywhere in the filesystem.