Bazel excessively re-fetches external repositories

[AlessandroPatti]

Description of the problem / feature request:

Since 2.2.0, Bazel fetches external repository more frequently. The culprit seems to be a3a2215, as the STARLARK_SEMANTIC value stored in the marker file changes with the commands.

Bugs: what's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

I am able to constantly reproduce it executing a run or build command followed by a query command.

What operating system are you running Bazel on?

MacOS 10.15.3

What's the output of bazel info release?

release 2.2.0

[laurentlb]

This might happen if you have different flags for build and for query.
Do you have a bazelrc file? If so, there might be an issue there.

If you have:

build --some_flag

You can enable it for query too. "common" means both "build" and "query", so you can try:

common --some_flag

[AlessandroPatti]

Syncing all flags in StarlarkSemanticsOptions.java using common did solve the issue indeed, thanks!

[keith]

I can confirm this fixed it for us too, in our case it was --incompatible_allow_tags_propagation.

There's more conversation about this here too https://bazelbuild.slack.com/archives/CA31HN1T3/p1583983830261700

But I wonder if we should force these flags to only ever be in a common group in the .bazelrc so no one else can accidentally make this mistake?

[aiuto]

Changing the flags passed to Bazel can force the reevaluation of the WORKSPACE. This is expected, but Bazel can spend a lot of time redownloading git repositories.

When I run bazelisk --migrate test ... in Stardoc, the command takes 15 minutes. Almost all the time is spent fetching and refetching git repositories.

Small repro:

WORKSPACE:

load("@bazel_tools//tools/build_defs/repo:git.bzl", "git_repository")

git_repository(
    name = "io_bazel",
    commit = "b2b9fa00d7d168e9553cfc9fe381928e8e246176",  # 2020-07-28
    remote = "https://github.com/bazelbuild/bazel.git",
    shallow_since = "1595950714 -0700",
)

BUILD:

load("@io_bazel//:distdir.bzl", "distdir_tar")

Example commands to run:
Copied from #11864

$ bazel build ...
$ bazel build ... --record_rule_instantiation_callstack
$ bazel build ... --record_rule_instantiation_callstack --incompatible_restrict_string_escapes

Most of the incompatible flags force a redownload. Each of the command spends around one minute downloading io_bazel, according to the console output:

     Fetching @io_bazel; Cloning b2b9fa00d7d168e9553cfc9fe381928e8e246176 of https://github.com/bazelbuild/bazel.git 58s

[jin]

There's a new report filed at #11895.

[TheButlah]

Same issue here, except it doesn't appear as though my build and query commands can be unified to use the same arguments, so the workaround given doesn't help in my case.

Bazelrc File:

# Allows for build:macos syntax
common --enable_platform_specific_config

# Build with clang-11 by default on linux (mac uses xcode clang)
build:linux --repo_env=CXX=clang++-11 --repo_env=CC=clang-11

# Clang is missing the global sized delete operator, but gcc has it by default. Make
# clang also use it same as gcc.
build --cxxopt="-fsized-deallocation"

# Timeout after 5 seconds for remote
build --remote_timeout=5
fetch --remote_timeout=5

# Fixes issue where debug symbols point to sources in deleted sandbox, by not deleting the sandbox
build:macos --sandbox_debug #--spawn_strategy=local

# Ensures warnings are always printed out, even when cached
build --experimental_replay_action_out_err

# Be more aggressive about avoiding questionable code
build --cxxopt="-Wall"

I've gone through them and they don't appear to be valid commands for query, unless I missed something. In this case, is there a fix or workaround? What is the source of my woes?

This is a big issue for my teams workflow because we use bazel query every time we import a python package in order to properly import built artifacts, and re-fetching dependencies each time slows things down a large amount

[laurentlb]

Can you identify which flag causes the problem?

Bazel will do a reload if a flag (that affects the loading phase) changes. If you identify which flag is problematic, we can look for a solution (it's possible that a Bazel flag doesn't behave well; or it can be a problem in your bazelrc).

[TheButlah]

@laurentlb it is the bazel build --repo_env that causes the fetch/query thrashing. Both bazel query and bazel fetch do not have a --repo_env flag, but its presence in bazel build causes bazel to consider the fetch and build steps to be different, hence the thrashing.

[laurentlb]

Thanks, I've filed a separate issue for this.

[igorgatis]

Is there a way to spot the flag which is causing this issue? Say, comparing logs?

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 14 days unless any other activity occurs or one of the following labels is added: "not stale", "awaiting-bazeler". Please reach out to the triage team (@bazelbuild/triage) if you think this issue is still relevant or you are interested in getting the issue resolved.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please reach out to the triage team (@bazelbuild/triage). Thanks!