Request: async commands

[rnikander]

I've been writing some scripts to generate a build.ninja file for a C++ project that uses C++ modules, with Clang. You need to run clang-scan-deps on a source file, and it tells you about the file's module dependencies. It gives you the symbolic names of the modules imported by, and provided by, the scanned file. You won't know what source files provide these required modules, until the build process gets to them and runs clang-scan-deps on them in turn.

As you process each source file, you can imagine adding rows to a table that looks like:

c++ file	pcm file	provides	requires
a/b/foo.cppm	build/a/b/foo.pcm	foomod	bar, baz
other/Bar.cppm	build/other/Bar.pcm	bar	bar:part1, mod3, mod4
stuff.cpp	stuff.pcm	baz	...
...

Eventually you have sufficient data - something like the transitive closure of required modules - that you can compile a file like foo.cppm.

If I write ninja code like:

build foo.pcm : precompile foo.cppm | foo.pcm.mod_deps
   dyndep = foo.pcm.mod_deps
build foo.pcm.mod_deps : get_mod_deps foo.cppm
rule get_mod_deps
   command = python3 get_mod_deps.py $in $out ...

My script there (get_mod_deps.py) needs to find a way to cooperate with the other scans and wait for data in the table. I know how to write that, but what happens if Ninja starts a bunch of these processes and they are all just hanging? That's not good. If it's got a limit, it could run out. Or it could flood the system with processes.

What I think I want here is some kind of asynchronous command. Instead of a thread getting blocked, the Ninja command could run and immediately return, but the rule would wait for an async callback of some kind to tell it that the command was "done". I can imagine a few ways to implement this. Maybe a socket that listens for simple strings like "ID:2535:Done".

Eventually a run of get_mod_deps.py would complete some information and it could send "Done" signals to one or more waiting Ninja commands.

What do people think? Would it be useful? Is there a way to do this already?

1. Clang's C++ module docs