Build *just* generated source files--for editor integrations

[cpsauer]

Description of the feature request:

Hey wonderful bazelers,

It's be awesome if Bazel had a flag to run a build--but, rather than executing all the actions, just run those needed to generate all the source files, or, e.g., all headers potentially used by source files in the main workspace.

The underlying problem is that autocomplete in most editors--as well as static analysis tools, require indexing the source, which leads to errors before all the included headers have been generated. (One example, of many.) Waiting until a full build has been completed is slower than some users would like on large projects. And commands like, say, bazel build $(bazel query 'filter(".*\.(?:hpp|inl|h|hh|hxx)$", kind("generated file", //...:*))') don't take configuration into account, and therefore don't work with, e.g. cross-compilation. If there is already a way to do this (or a duplicate issue) and I've missed it in my searching, my apologies, but I'd really appreciate it if you'd point me towards it.

Thanks so much,
Chris
(ex-Googler and bazel compile commands extractor author)

P.S. The "Work seamlessly with IDEs" part of #6862 seems semi relate to this, in that IDEs will likely want to pull down these files without necessarily kicking off a full build.

[brentleyjones]

👍 The amount of work that rules_xcodeproj has to do to support this, and it's not without bugs, is a lot.

One thing that might happen with this though, is that a generated source could be an output from a compilation (e.g. swift_library's generated header could be an input into a modulemap rule), which can cause a large chunk of the build graph to be compiled.

[SoftwareApe]

@brentleyjones I don't think anybody would complain if a large chunk of the build graph has to be compiled to generate a source, if that's what's in the generated source's dependency path. But a lot of projects have plenty of generated files that only depend on running some python scripts with minimal dependencies. Those could have substantially smaller build graphs being built for their generated files.

[tjgq]

Could you write an aspect that collects source files from the entire build graph into an output group, then ask Bazel to invoke the aspect on some appropriate top-level target(s) and build that output group?

For some languages, it's even possible that the complete transitive closure of source files is already available on some language-specific provider on the "binary" target, in which case the aspect wouldn't even need to deeply traverse the graph.

[brentleyjones]

That's exactly what rules_xcodeproj does. It's hard to determine exactly what are the sources though. Search for generated here and you will see just some of the complexity. And I'm sure I'm getting something wrong still.

[tjgq]

Having written aspects that look a bit like this in the past, I feel the pain.

I'm not convinced there's a way to meaningfully improve on the aspect approach, except having the rules themselves collect the transitive set of source files. It's even hard for me to imagine a Bazel feature that would help with this problem generally. Any solution based on a naive graph traversal won't be able to tell apart files that look like sources but aren't (e.g., a .cpp file in a data attribute) without incorporating rule-specific knowledge about the semantics of attributes.

[fmeum]

If a lightweight language-agnostic ruleset such as Skylib had a provider that any kind of language rule could use to indicate source files, collecting them with an aspect could be much simpler and less reliant in heuristics. Maybe that's something to think about together with the IntelliJ plugin devs?

Semi-related: https://bazelbuild.slack.com/archives/CA31HN1T3/p1677679466624889?thread_ts=1677679466.624889&cid=CA31HN1T3

[cpsauer]

Thanks for such a thoughtful discussion, all.
@brentleyjones, did you find that the aspect approach worked well with architecture-split configurations/transitions?

[coeuvre]

Also note that using aspect to collect outputs into an output group and building that output group is a preferred way (over regex) to ask Bazel download outputs when building without the bytes.

[brentleyjones]

@cpsauer Yes, it's the main reason I went the aspect route. And as Chi just mentioned, it's allowed downloading the files while using BwtB.

[meisterT]

cc @comius @lberki

[cpsauer]

Hey, all. Thanks again for being so excellent here.

I think we're hearing two themes.

1. You can do this with aspects, building the output group they create.
   [Apologies that I didn't lead with that. Had some out-of-date/bad ideas in my head about them. Thanks for your patience in setting me straight!]
2. The aspect-based approach comes with some pain--maybe there's an opportunity to improve things by letting people call into a shared implementation.

Folks, what to you think the best intermediate interface here might be--if there is a better one than the current? Some options spawned from the discussion:

• A regex flag that lets users quickly and easily specify which files they care about generating.
  [This'd be something like a generalization of --experimental_remote_download_regex for local builds and some variants of this might be able to solve its use cases? A key advantage of this one is that it'd let people really easily express that they want to rebuild only some sub-part of the configured graph, including, even, individual files they're opening. Alternate versions might reuse some bazel query language instead of a regex.]
• A provider as a convention, that rules are asked to expose to make editing easy, per Build *just* generated source files--for editor integrations #17660 (comment)
• Or aspects are really the best interface/enough. In that case we should close this down--my having wrongly not considered them from the start--but perhaps first seeing if there's a good place/venue to share implementations.

Offhand, the first of those seems pretty generally useful to me, but I know many of you all have gotten much deeper into this than me.

Cheers,
CS

[comius]

I think aspects can do this already. Triaging to P3: I'm working on some improvements on aspects so that you could get generated files a bit faster. I'm open to ideas that would make aspects approach easier.

Or maybe we should just provide some baseline aspect that does this, that people can call/extend to suit their needs?

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[SoftwareApe]

I think aspects can do this already. Triaging to P3: I'm working on some improvements on aspects so that you could get generated files a bit faster. I'm open to ideas that would make aspects approach easier.

Or maybe we should just provide some baseline aspect that does this, that people can call/extend to suit their needs?

Did you make any progress here? At least from my view this could use an example implementation in the docs at least, if it is possible.