Multiple agents

[krafs]

Liking the idea of the project! Found it when researching how to implement this kind of app myself :)

Question:
Could ModularPipelines be made to distribute its parallelizable work on multiple agents, instead of running it all on one agent?

[thomhurst]

Thanks @krafs !

Currently this isn't an out of the box feature for automatic splitting of work, and would be a substantial bit of work for the different processes/machines to be able to communicate and send/receive module results to each other.

But if you can work out manually an efficient way to split up your jobs, then that's possible.

You could pass in some sort of environment variable to the program E.G. JobCategory=Test and another JobCategory=Build and then either:

In the startup, check that env variable and conditionally register modules

Or

In a module, override the ShouldSkip method and skip it based on a condition such as your environment variable.

Then in your GitHub actions yaml file (or other build system) just create two or more separate jobs to run your modular pipeline program, each time passing in different variables

The only thing to consider is that if a module was run on another agent, then you won't be able to wait for it or get it's results on a different agent.

[krafs]

The way I imagined making this myself there'd be an initial agent that builds the host and then dynamically creates jobs via a matrix strategy. And then yes - invoking the host with different "endpoints" to run different things on the different hosts. And then synchronize work and data via job outputs and artifacts storage.

You're right, it is probably a lot of work. I find the idea very compelling though, so I might give it a go :)

[thomhurst]

It would would be great in theory but needs a way to also be build system agnostic. So artifacts could work for GitHub, but we'd also need to consider other build systems.

Also there isn't a public endpoint for uploading artifacts is there, so we couldn't upload as we go from C#? Afaik it's only a GitHub action, and that means having to wait until the dotnet process has ended, which is a bit of a bottleneck.

[krafs]

Yeah, there's no completely agnostic way of sharing data via artifact storage. It'd have to be vendor-specific.

I think the GitHub artifacts action is just using @actions/artifacts for interacting with the artifacts storage, along with the GITHUB_TOKEN. I don't see why that couldn't be re-implemented in C#. But receiving agents would probably have to resort to polling the storage for changes, otherwise they'd have to wait for the first job to complete so it can trigger on the output.