Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17358

coeuvre · 2023-01-30T16:38:21Z

…g an invocation

Part of #16660.

brentleyjones · 2023-01-30T16:58:29Z

src/main/java/com/google/devtools/build/lib/remote/RemoteActionInputFetcher.java

-          IOException annotatedException =
-              new IOException(
-                  String.format(
-                      "Failed to fetch file with hash '%s' because it does not"


I know people use the hash from the error message for debugging. I also know they can look at the gRPC log to get the same hash, but not everyone collects that all the time. Are we able to preserve the hash in the error?

The hash is saved in original BulkTransferException exception and will be printed out. e.g.

Failed to fetch blobs because they do not exist remotely. --remote_download_outputs=minimal does not work if your remote cache evicts blobs during builds: Missing digest: b94d27b9934d3e08a52e52d7da7dabfac484efe37a5380ee9088f7ace2efcde9/11

Updated test to assert on the hash.

…g an invocation

coeuvre · 2023-02-13T13:23:17Z

As discussed internally, I added flag --incompatible_remote_use_new_exit_code_for_lost_inputs (in the internal branch) to make this change backwards compatible. I will flip this flag right away after this PR is merged in main branch and let users in 6.x branch have the control over the behaviour.

coeuvre · 2023-02-14T13:36:45Z

The flag is flipped in c77790f.

coeuvre · 2023-02-14T13:37:30Z

@bazel-io fork 6.1.0

…g an invocation Part of #16660. Closes #17358. PiperOrigin-RevId: 509494072 Change-Id: Id6944da5d9a556dc9154fcb702948586b474875e

…g an invocation Part of bazelbuild#16660. Closes bazelbuild#17358. PiperOrigin-RevId: 509494072 Change-Id: Id6944da5d9a556dc9154fcb702948586b474875e

…g an invocation (#17496) Part of #16660. Closes #17358. PiperOrigin-RevId: 509494072 Change-Id: Id6944da5d9a556dc9154fcb702948586b474875e Co-authored-by: kshyanashree <109167932+kshyanashree@users.noreply.github.com> Co-authored-by: keertk <110264242+keertk@users.noreply.github.com>

With #17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With #17462 and #17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with #17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of #16660. Closes #17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

With bazelbuild#17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With bazelbuild#17462 and bazelbuild#17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with bazelbuild#17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of bazelbuild#16660. Closes bazelbuild#17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

…ror (#18171) With #17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With #17462 and #17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with #17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of #16660. Closes #17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

With bazelbuild#17358, Bazel will exit with code 39 if remote cache evicts blobs during the build. With bazelbuild#17462 and bazelbuild#17747, Bazel is able to continue the build without bazel clean or bazel shutdown. However, even with bazelbuild#17639 and following changes to extend the lease, remote cache can still evict blobs in some rare cases. Based on above changes, this PR makes bazel retry the invocation if it encountered the remote cache eviction error during previous invocation if `--experimental_remote_cache_eviction_retries` is set, or **build rewinding**. ``` $ bazel build --experimental_remote_cache_eviction_retries=5 ... INFO: Invocation ID: b7348bfa-9446-4c72-a888-0a0ad012f225 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt ERROR: .../workspace/a/BUILD:8:8: Executing genrule //a:bar failed: Failed to fetch blobs because they do not exist remotely: Missing digest: b5bb9d8014a0f9b1d61e21e796d78dccdf1352f23cd32812f4850b878ae4944c/4 Target //a:bar failed to build Use --verbose_failures to see the command lines of failed build steps. INFO: Elapsed time: 0.447s, Critical Path: 0.05s INFO: 2 processes: 2 internal. ERROR: Build did NOT complete successfully Found remote cache eviction error, retrying the build... INFO: Invocation ID: 983f60dc-8bb9-4b82-aa33-a378469ce140 Loading: Loading: Loading: 0 packages loaded Analyzing: target //a:bar (0 packages loaded, 0 targets configured) INFO: Analyzed target //a:bar (0 packages loaded, 0 targets configured). INFO: Found 1 target... [0 / 2] [Prepa] BazelWorkspaceStatusAction stable-status.txt Target //a:bar up-to-date: bazel-bin/a/bar.out INFO: Elapsed time: 0.866s, Critical Path: 0.35s INFO: 3 processes: 1 internal, 1 processwrapper-sandbox, 1 remote. INFO: Build completed successfully, 3 total actions $ ``` Part of bazelbuild#16660. Closes bazelbuild#17711. PiperOrigin-RevId: 520610524 Change-Id: I20d43d1968767a03250b9c8f8a6dda4e056d4f52

coeuvre requested review from a team, gregestren and fweikert as code owners January 30, 2023 16:38

coeuvre requested review from tjgq and removed request for a team January 30, 2023 16:38

brentleyjones reviewed Jan 30, 2023

View reviewed changes

ShreeM01 added team-Remote-Exec Issues and PRs for the Execution (Remote) team awaiting-user-response Awaiting a response from the author labels Jan 31, 2023

coeuvre added awaiting-review PR is awaiting review from an assigned reviewer and removed awaiting-user-response Awaiting a response from the author labels Jan 31, 2023

coeuvre requested a review from brentleyjones January 31, 2023 09:37

brentleyjones approved these changes Jan 31, 2023

View reviewed changes

coeuvre added 2 commits February 6, 2023 17:37

Exit with code 39 if remote cache evicted blobs that Bazel need durin…

68321cb

…g an invocation

Fix tests

0506b21

coeuvre force-pushed the lease-service-0 branch from 4615d1a to 0506b21 Compare February 6, 2023 16:41

copybara-service bot closed this in 99cff33 Feb 14, 2023

coeuvre deleted the lease-service-0 branch February 14, 2023 13:36

bazel-io mentioned this pull request Feb 14, 2023

[6.1.0] Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17485

Closed

hvadehra pushed a commit that referenced this pull request Feb 14, 2023

Exit with code 39 if remote cache evicted blobs that Bazel need durin…

7c02dca

…g an invocation Part of #16660. Closes #17358. PiperOrigin-RevId: 509494072 Change-Id: Id6944da5d9a556dc9154fcb702948586b474875e

coeuvre mentioned this pull request Feb 15, 2023

Retry invocation if Bazel exits with code 39 bazelbuild/bazelisk#425

Closed

coeuvre mentioned this pull request Feb 15, 2023

[6.1.0] Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17496

Merged

coeuvre mentioned this pull request Mar 9, 2023

Automatically retry the build if encountered remote cache eviction error #17711

Closed

ShreeM01 mentioned this pull request Mar 30, 2023

[6.2.0]Automatically retry the build if encountered remote cache eviction error #17938

Closed

ShreeM01 mentioned this pull request Apr 3, 2023

[6.2.0]Automatically retry the build if encountered remote cache eviction error #17953

Closed

coeuvre mentioned this pull request Apr 21, 2023

[6.2.0] Automatically retry the build if encountered remote cache eviction error #18171

Merged

coeuvre mentioned this pull request Sep 7, 2023

New exit code for remote cache errors #16513

Closed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17358

Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17358

coeuvre commented Jan 30, 2023

brentleyjones Jan 30, 2023

coeuvre Jan 31, 2023

coeuvre commented Feb 13, 2023 •

edited

Loading

coeuvre commented Feb 14, 2023

coeuvre commented Feb 14, 2023

Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17358

Exit with code 39 if remote cache evicted blobs that Bazel need durin… #17358

Conversation

coeuvre commented Jan 30, 2023

brentleyjones Jan 30, 2023

Choose a reason for hiding this comment

coeuvre Jan 31, 2023

Choose a reason for hiding this comment

coeuvre commented Feb 13, 2023 • edited Loading

coeuvre commented Feb 14, 2023

coeuvre commented Feb 14, 2023

coeuvre commented Feb 13, 2023 •

edited

Loading