Add profiling for Bzlmod operations

[fmeum]

The resolution of Bazel modules and evaluation of module extensions is now represented in the profile.

Profile from a run of bazel build //src:bazel-dev --enable_bzlmod --nobuild:

[fmeum]

The profile shows that module resolution is much slower than it could be due to serial downloads. Maybe we can get a quick win here with a thread pool and then adopt Loom later on?

[meteorcloudy]

@meisterT Can anyone from the build productivity team also take a look?

[meteorcloudy]

Awesome, thanks!

[Wyverald]

thank you!

[Wyverald]

Could you share your profile result here? I'd love to take a look too (too many '...'s in that screenshot :))

[fmeum]

Could you share your profile result here? I'd love to take a look too (too many '...'s in that screenshot :))

command.profile.gz

[fmeum]

It's interesting to see how the cyclic dependency between rules_go and gazelle contributes to the critical path.

[Wyverald]

It's interesting to see how the cyclic dependency between rules_go and gazelle contributes to the critical path.

Yeah that's essentially a really long dependency chain. We could be smarter there -- don't bother with deps at lower versions if we know there's no multiple-version override, for example. It would make the discovery code a bit more gnarly, but anyway.

[Wyverald]

It's interesting to see how the cyclic dependency between rules_go and gazelle contributes to the critical path.

Yeah that's essentially a really long dependency chain. We could be smarter there -- don't bother with deps at lower versions if we know there's no multiple-version override, for example. It would make the discovery code a bit more gnarly, but anyway.

But do note that this needs to be done very carefully, or version resolution might become nondeterministic-- if the lower version itself has higher-version deps, we need to make sure we have a deterministic way to decide which "lower version" to drop. The end result is still some set of minimal versions, but is not the original MVS. This is related to the original idea by @meteorcloudy

[fmeum]

(didn't mean to rerequest a review, I didn't make any changes)

[fmeum]

@bazel-io flag

[iancha1992]

@bazel-io fork 6.4.0

[iancha1992]

@fmeum @meisterT @Wyverald @meteorcloudy I think we may need another commit before we cherry-pick this one to 6.4.0 because in src/main/java/com/google/devtools/build/lib/runtime/BlazeCommandDispatcher.java line 622, we have earlyExitCode = optionHandler.parseStarlarkOptions(env, reporter); in the release-6.4.0 branch. But it really should be earlyExitCode = optionHandler.parseStarlarkOptions(env); before the change.

cc: https://github.com/orgs/bazelbuild/teams/triage

[iancha1992]

The changes in this PR have been included in Bazel 6.4.0 RC1. Please test out the release candidate and report any issues as soon as possible. If you're using Bazelisk, you can point to the latest RC by setting USE_BAZEL_VERSION=last_rc.
Thanks!