Make system module map generation faster and fully hermetic

[fmeum]

• system_module_maps no longer performs any IO.
• The generated module map no longer references any non-hermetic paths and can thus be cached remotely, even when toolchain or sysroot are provided as absolute paths.

[siddharthab]

Please land whenever you are ready. Let's make a release when this lands.

[honnix]

Could you please help me understand how to make RBE work in this case? For example when running bazel from macOS and executing in a remote Linux executor, /usr/include does not exist on macOS but exists on Linux, this filter will remove /usr/include from cxx_builtin_include_directories and that could result in bazel validation failure ...includes the following non-builtin files with absolute paths (if these are builtin files, make sure these paths are in your toolchain).

[fmeum]

I didn't have this scenario in mind when I wrote this, my assumption being that you would only use this toolchain for RBE together with a hermetic sysroot.

This function only makes sense if you are actually targetting the host, so we should probably disable it when cross compiling and just use all directories unfiltered. If you want to give that a try, I can help you implement and test this approach.

[honnix]

I see. I'm not sure of a reasonable implementation. What would you suggest?

[fmeum]

The _is_hermetic_or_exists function should be fed the information whether exec_os equals target_os and only perform the existence check if so.

[honnix]

Yeah I was looking at those, but not sure that is reliable. For RBE, exec_os could be the same as target_os, right? For example I can define a toolchain like the following and while produce target for linux as well.

llvm.toolchain(
    name = "llvm_toolchain_linux_exec",
    exec_arch = "amd64",
    exec_os = "linux",
    llvm_version = LLVM_VERSION,
)

[fmeum]

You are completely right, I guess we need to match the host OS (obtained e.g. from repository_ctx.os) against the target OS (this determines the shape of the sysroot).

[honnix]

Let me give that a try.

[honnix]

rctx.os.name for mac is like mac os x, which is not the same as the convention used by target_os :( But otherwise this should be a reasonable approach. We just need to do some formalisation.