Register synthetic targets for subsystems.

[kaos]

This is the first step towards having proper dependencies to subsystem configurations.

Planned next step is to add a dependency on synthesized configuration targets based on option value sources per subsystem.

Can eventually be used to address things like #14724

Example config source info as available already:

This adds quite a number of targets to the root, which can easily be filtered out however:

╰─❯ ./pants list //
//:buildgrid_remote
//:docker_env
//:subsystem-anonymous-telemetry
//:subsystem-auth
//:subsystem-auth-acquire
//:subsystem-auth-token-check
//:subsystem-auth-token-info
//:subsystem-autoflake
//:subsystem-black
...

╰─❯ ./pants list --filter-target-type=-_subsystem //
//:buildgrid_remote
//:docker_env
//BUILD_ROOT:files
//cargo:scripts
//conftest.py:test_utils
//pants:scripts
//pants.toml:files
╰─❯ 

[stuhood]

Sorry that I haven't reviewed this yet: will check it out by EOD.

[Eric-Arellano]

Interesting idea. How will the dependency rules be implemented?

For --changed-since, are you trying to get changes in the subsystem to result in changes to impacted targets to show up? How would we know a subsystem has changed? We don't have access to the prior values of the options

[kaos]

Interesting idea. How will the dependency rules be implemented?

After adding synthetic targets for config files, such as pants.toml and others,
my idea was to look at the config sources for each option value, and have the subsystem depend on each of those sources. Initially crude, I'm sure, but we may refine from that. (as an improvement over today where we have no dependency at all from a subsystem to its config sources)

Hmm, this gets me thinking, aren't the subsystem values part of the hash value, so if there is a config change that would already invalidate the graph depending on that subsystem?

For --changed-since, are you trying to get changes in the subsystem to result in changes to impacted targets to show up? How would we know a subsystem has changed? We don't have access to the prior values of the options

Similar here I think, rather crude that if you make any change in pants.toml for instance, then all subsystems that have any value coming from that file will be invalidated. Just to get started some where.

[stuhood]

Hmm, this gets me thinking, aren't the subsystem values part of the hash value, so if there is a config change that would already invalidate the graph depending on that subsystem?

Only at runtime: that's the reasoning behind runtime graph awareness: #11206

[kaos]

Hmm, this gets me thinking, aren't the subsystem values part of the hash value, so if there is a config change that would already invalidate the graph depending on that subsystem?

Only at runtime: that's the reasoning behind runtime graph awareness: #11206

Oh.. ok. Let me see if I understand this correctly. So, this would mean that if we invalidate target x due to a change in pants.toml then if that change didn't affect subsystem y, we'll get the cached result from a previous run?

If so, I feel that it's less bad to have these coarse invalidations going, as it may report more targets as changed, but the actual work performed will be correct.

[Eric-Arellano]

The effect of what you're describing is that adding a new blank line to pants.toml will invalidate almost every single target, since most targets have a relationship to subsystems. I'm skeptical we want that so coarse grained.

[kaos]

The effect of what you're describing is that adding a new blank line to pants.toml will invalidate almost every single target, since most targets have a relationship to subsystems. I'm skeptical we want that so coarse grained.

It would only invalidate for those subsystems you have configuration for in your pants.toml, and that is likely not every subsystem out there.

Also, more often than not, the change you make will be relevant to some part of the project, rather than simply being just a formatting change, so to me it would make sense to invalidate a lot in order to pick up the change properly rather than not at all. This is no worse than when you regenerate your lockfile and it gives you a pretty big blast radius.

[Eric-Arellano]

It would only invalidate for those subsystems you have configuration for in your pants.toml, and that is likely not every subsystem out there.

How so? How will you know the subsystem is defined in pants.toml? Parse the TOML?

[kaos]

It would only invalidate for those subsystems you have configuration for in your pants.toml, and that is likely not every subsystem out there.

How so? How will you know the subsystem is defined in pants.toml? Parse the TOML?

It's in the PR description screen shot and text ;)

(and further discussed in my comment here #17653 (comment) )

[stuhood]

Hmm, this gets me thinking, aren't the subsystem values part of the hash value, so if there is a config change that would already invalidate the graph depending on that subsystem?

Only at runtime: that's the reasoning behind runtime graph awareness: #11206

Oh.. ok. Let me see if I understand this correctly. So, this would mean that if we invalidate target x due to a change in pants.toml then if that change didn't affect subsystem y, we'll get the cached result from a previous run?

No: it would mean that we'd be able to precisely report the files that are consumed in order to run, say, test, without the need for synthetic targets to be generated for them. Targets would just represent metadata, rather than "the unit of dependency", per-se: the unit of dependency (at least for the purposes of --changed) would be a file instead.

we'll get the cached result from a previous run

Actual cache keys would be unaffected by #11206, same as they are not affected here: the cache key is still based on what @rules request at runtime, and which Process structs they construct. #11206 (and the synthetic targets work) are both about making --changed match what @rules actually request... but #11206 proposes to try and do it by introspecting what they do request, rather than by creating synthetic targets to correspond to each file that is consumed.

[kaos]

Closing as stale.