Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Use ctime in file digest cache key #18115

Merged
merged 1 commit into from
Apr 17, 2023
Merged

Use ctime in file digest cache key #18115

merged 1 commit into from
Apr 17, 2023

Conversation

fmeum
Copy link
Collaborator

@fmeum fmeum commented Apr 17, 2023 •
edited
Loading

File digests are now additionally keyed by ctime for supported file system implementations. Since Bazel has a non-zero default for --cache_computed_file_digests, this may be required for correctness in cases where different files have identical mtime and inode number. For example, this can happen on Linux when files are extracted from a tar file with fixed mtime and are then replaced with mv, which preserves inodes.

Since Java (N)IO doesn't have support for reading file ctimes on Windows, a new method backed by a native implementation is added to WindowsFileOperation. Adding a call to this function to stat uncovered previously silent bugs where Unix-style PathFragments were created on Windows:

  1. Bzlmod's createLocalRepoSpec did not correctly extract the path from a registry's file:// URI on Windows.
  2. --package_path isn't usable with absolute paths on Windows as it splits on :. Since the flag is deprecated, this commit fixes the tests rather than the implementation.

(Bzlmod change was dropped when backporting to 5.4.1 because the createLocalRepoSpec function did not exist in that release yet)

Fixes #14723

Closes #18003.

PiperOrigin-RevId: 524297459
Change-Id: I96bfc0210e2f71bf8603c7b7cc5eb06a04048c85

@fmeum
Use ctime in file digest cache key
7e1719d 
File digests are now additionally keyed by ctime for supported file system implementations. Since Bazel has a non-zero default for `--cache_computed_file_digests`, this may be required for correctness in cases where different files have identical mtime and inode number. For example, this can happen on Linux when files are extracted from a tar file with fixed mtime and are then replaced with `mv`, which preserves inodes.

Since Java (N)IO doesn't have support for reading file ctimes on Windows, a new method backed by a native implementation is added to `WindowsFileOperation`. Adding a call to this function to `stat` uncovered previously silent bugs where Unix-style `PathFragment`s were created on Windows:

1. Bzlmod's `createLocalRepoSpec` did not correctly extract the path from a registry's `file://` URI on Windows.
2. `--package_path` isn't usable with absolute paths on Windows as it splits on `:`. Since the flag is deprecated, this commit fixes the tests rather than the implementation.

Fixes bazelbuild#14723

Closes bazelbuild#18003.

PiperOrigin-RevId: 524297459
Change-Id: I96bfc0210e2f71bf8603c7b7cc5eb06a04048c85
@fmeum fmeum mentioned this pull request Apr 17, 2023
Closed
@keertk keertk mentioned this pull request Apr 17, 2023
Closed
@keertk
Copy link
Member

keertk commented Apr 17, 2023

@fmeum confirming that this isn't a work in progress? Can I go ahead and merge once presubmits pass?

@meteorcloudy
Copy link
Member

@fmeum I guess the change is not exactly the same as the one in the main branch? Can you explain a bit about what you had to change to resolve the conflict?

@fmeum
Copy link
Collaborator Author

fmeum commented Apr 17, 2023

@meteorcloudy I dropped the Bzlmod change as local repo registry soures weren't available in Bazel 5. I also had to modify https://github.com/bazelbuild/bazel/pull/18115/files#diff-0e04f955a8b8a9aa27a09d6528db2432267b7f6a42c897efcba46708a6c4f39bR669 since the experimental feature for writable outputs also isn't part of Bazel 5.

Should I update the commit message to remove the mention of Bzlmod?

@fmeum fmeum marked this pull request as ready for review April 17, 2023 15:32
@fmeum fmeum requested a review from ShreeM01 as a code owner April 17, 2023 15:32
@meteorcloudy
Copy link
Member

I see, I guess it's not easy to keep the original commit and apply the fixes as separate commit? Just want to make sure it's easy to tell what was changed in case of any problem.

Should I update the commit message to remove the mention of Bzlmod?

I guess it's better to add additional message like "Bzlmod change was dropped when backporting to 5.4.1 because relevant feature wasn't available"?

@fmeum
Copy link
Collaborator Author

fmeum commented Apr 17, 2023

@meteorcloudy I updated the PR description. I don't think splitting this into a pure cherry-pick and a fixup is possible. Some methods, including one that is touched by the cherry-pick, have been renamed since 5.4.1.

@meteorcloudy
Copy link
Member

@fmeum SG, thanks!

@keertk keertk enabled auto-merge (squash) April 17, 2023 16:30
@keertk keertk merged commit a35f592 into bazelbuild:release-5.4.1 Apr 17, 2023
@fmeum fmeum deleted the release-5.4.1-14723 branch April 17, 2023 16:56
@keertk keertk added team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file. team-Core Skyframe, bazel query, BEP, options parsing, bazelrc labels Apr 17, 2023
copybara-service bot pushed a commit that referenced this pull request Apr 19, 2023
@keertk
Release 5.4.1 (2023-04-19)
0210e45 
Baseline: 8d66a41

Cherry picks:

   + becd149:
     Remote: Cache merkle trees
   + d7628e1:
     Update DEFAULT_IOS_CPU for M1 arm64 simulator support
   + 80c56ff:
     Compile Apple tools as fat binaries if possible
   + 3c09f34:
     Add protobuf as a well known module
   + 3a5b360:
     Remote: Merge target-level exec_properties with
     --remote_default_exec_properties
   + 917e15e:
     Add -no_uuid for hermetic macOS toolchain setup
   + f5cf8b0:
     Remote: Fixes an issue when --experimental_remote_cache_async
     encounter flaky tests.
   + 77a002c:
     Remove DigestUtils.getDigestInExclusiveMode() now that SsdModule
     has …
   + 557a7e7:
     Fixes for the Starlark transition hash computation (#14251)
   + 34c7146:
     Do location expansion in copts of objc_library
   + 50274a9:
     [5.x] Remote: Add support for compression on gRPC cache (#14277)
   + 61bf2e5:
     Automated rollback of commit
     34c7146.
   + 79888fe:
     Silence a zstd-jni GCC warning.
   + 063b5c9:
     Remote: Limit max number of gRPC connections by
     --remote_max_connections.
   + fd727ec:
     Do location expansion in copts of objc_library
   + 23d0969:
     Fix _is_shared_library_extension_valid
   + 5cf1d6e:
     Remove merging of java_outputs in JavaPluginInfo.
   + cea5f4f:
     Cherrypick Bzlmod documentation (#14301)
   + 227e49e:
     Format work requests according to ndjson spec
   + ae0a6c9:
     Enable user_link_flags_feature for macosx cc_toolchain_config
   + 8c2c78c:
     Remote: Use Action's salt field to differentiate cache across
     workspaces.
   + f948989:
     [5.x] Remote: Fix "file not found" error when remote cache is
     changed from enabled to disabled.  (#14321)
   + 3069ac4:
     Delete marker file before fetching an external repository
   + c05c626:
     Remote: Fix file counting in merkletree.DirectoryTreeBuilder
   + d84f799:
     Fix remote spawn tests for remote_merkle_tree_cache=true
   + 59e16e9:
     Show skipped tests as a warning
   + 76b3c24:
     Build xcode-locator as a universal binary
   + aa52f2d:
     Exit collect_coverage.sh early if LCOV_MERGER is not set.
   + 4256d46:
     Automated rollback of commit
     d84f799.
   + dce2435:
     [apple] fix issues compiling C in objc_library for watchos/armv7k
   + bfc2413:
     5.x: Remote: Ignore blobs referenced in BEP if the generating
     action cannot be cached remotely. (#14389)
   + 5aef53a:
     Remote: Don't blocking-get when acquiring gRPC connections.
     (#14420)
   + 005361c:
     Disable IncludeValidation for ObjC in bazel
   + d703b7b:
     Update java_tools v11.6
   + 90965b0:
     Stop remote blob upload if upload is complete. (#14467)
   + dc59d9e:
     [5.x] Make remote BES uploader better (#14472)
   + 2edab73:
     Avoid too verbose warnings in terminal when cache issues
   + 1160485:
     Rename --project_id to --bes_instance_name
   + c63d9ec:
     Automated rollback of commit
     bfdfa6e.
   + b341802:
     [apple] support watchos_arm64 in toolchain
   + 43bcf80:
     Disable implicitly collecting baseline coverage for toolchain
     targets.
   + 302971e:
     Automated rollback of commit
     7d09b4a.
   + 6200202:
     Bzlmod: Starlarkify default attr values for TypeCheckedTags
   + 38117d4:
     Fix build after rc4 cherrypicks (#14581)
   + 41feb61:
     Release 5.0.0 (2022-01-19)
   + 486d153:
     Find runfiles in directories that are themselves runfiles
   + 0de7bb9:
     Don't resolve symlinks for --sandbox_base
   + 8b60c90:
     Remove uses of -lstdc++ on darwin
   + 60f757c:
     Allow Label instances as keys in select (#14755)
   + 3836ad0:
     Remote: Only waits for background tasks from remote execution.
   + 8734ccf:
     Add the default solib dir to the rpath for cc_imports with
     transitions
   + 9e16a64:
     Flip --experimental_worker_allow_json_protocol
   + fce7ea8:
     Fix `ctx.fragments.apple.single_arch_cpu` returning incorrect
     cpu for tools when host cpu and exec cpu are different
   + 0c1d09e:
     Propagate --experimental_cc_implementation_deps to host config
   + 1c3a245:
     Support select() on constraint_value for aliases.
   + 67a133b:
     Improve documentation for select()
   + 5356fed:
     Cherrypicks for experimental cc_shared_library (#14773)
   + ffdd633:
     [apple] support tvos_sim_arm64 in toolchain (#14779)
   + a58ddea:
     Cherry pick win arm64 (#14794)
   + dc41a20:
     [5.1.0] cherrypick subpackages support (#14780)
   + 86e2db7:
     Add a helper method for rules to depend on the cpp toolchain
     type.
   + 6990c02:
     UrlRewriter should be able to load credentials from .netrc
     (#14834)
   + 32d1606:
     Add "arch" struct field to repository_os
   + 2cfdcea:
     [5.x] bzlmod: Add support for WORKSPACE.bzlmod (#14813)
   + c2ddbd1:
     Ignore missing include directory in JDK distribution.
   + 16de035:
     Fix bazel coverage false negative
   + 0c74741:
     Remote: Postpone the block waiting in `afterCommand` to
     `BlockWaitingModule` (#14833)
   + 3297d92:
     Switch to `ProcessHandle` for getting the PID (#14842)
   + a987b98:
     Fix uses of std++ on bsd
   + d184e48:
     Remote: handle early return of compressed blobs uploads
   + 0b09e9e:
     Add removeprefix/removesuffix to Starlark strings
   + d42ab0c:
     Fix default CPU for macOS and iOS (#14923)
   + cd24f39:
     Add paramfile support for def_parser, since in rare cases on
     Windows command line character limit was reached.
   + 0b1beef:
     Normalize rpath entries to guard against missing default solib
     dir
   + 24e8242:
     Fix aggressive params file assumption
   + c45838b:
     Fix precompiled libs not in runfiles of cc_shared_library
     (#14943)
   + 764614e:
     Bzlmod: Allow multiple `use_extension`s on the same extension
     (#14945)
   + fa761f8:
     Fix typo in `apple_common.platform` docs
   + f7d8288:
     Yield a Proxy for addresses without protocol
   + 8cefb8b:
     Avoid merging URLs in HttpUtils
   + b480480:
     Make protocOpts() public. (#14952)
   + 113eaca:
     Do not hide BulkTransferException messages when there were more
     than one exception
   + b1bf9d6:
     merkle_tree_cache: change default size to 1000
   + f15e0c7:
     Add --experimental_repository_cache_urls_as_default_canonical_id
     to help detect broken repository URLs (#14989)
   + f421474:
     Expose the logic to read user netrc file
   + b858ec3:
     Correct cpu and os values of `local_config_cc_toolchains` targets
   + 5e79972:
     Expose CoverageOutputGenerator on a Fragment (#14997)
   + 78f0311:
     Correct error runfiles cc_shared_library (#14998)
   + 7937dd1:
     [5.1] Adding Starlark dependencies to the package //external
     (#14991)
   + a73aa12:
     Remote: Fix crashes with InterruptedException when using http
     cache.
   + f8707c0:
     Account for interface libraries in cc_shared_library
   + a570f5f:
     Fix coverage runfiles directory issue
   + 95de355:
     Do not validate input-only settings in transitions (#15048)
   + 71747cc:
     Filter out system headers on macOS.
   + cb6500a:
     Update Bazel bootstrap documentation and remove obsolete flags.
     (#15065)
   + 4c031d1:
     [5.1] Undocument --bes_best_effort (#15066)
   + 267142f:
     Fix conflicting actions error when specifying
     --host_macos_minimum_os (#15068)
   + f192362:
     [5.1] Remote: Action should not be successful and cached if
     outputs were not created (#15071)
   + 00d74ff:
     Support decompressing zstd tar archives for repository rules.
   + f585783:
     Remote: Don't check TreeArtifact output
   + efb2b80:
     osx_cc_wrapper: Only expand existing response files
   + c771c43:
     Remote: Fix crashes by InterruptedException when dynamic
     execution is enabled. (#15091)
   + 3785677:
     Use python3 on macOS
   + 815d9e4:
     Release 5.1.0 (2022-03-24)
   + 1fbb69e:
     Prepare 5.1.1 release
   + df153df:
     Fix CODEOWNERS syntax
   + 2b92a31:
     Remote: Don't check declared outputs for failed action
   + b47aa71:
     Upgrade abseil version to the latest
   + c49c45d:
     Revert default export all symbols on Windows
   + 7d3fb99:
     Support ZIP files with total number of disks = 0
   + 0f5dc11:
     Release 5.1.1 (2022-04-08)
   + 2422cfb:
     Update CODEOWNERS
   + bbcff18:
     [5.2.0] Update java_tools 11.7.1 (#15231)
   + 9c98120:
     Add support for .ar archives (and .deb files)
   + d3435b0:
     Seperate GetSelfPath implementation for Blaze and Bazel
   + c94572b:
     Include jdk.crypto.mscapi in minimized Windows embedded JDK
   + 299022c:
     remote: Proactively close the ZstdInputStream in
     ZstdDecompressingOutputStream.
   + 2770799:
     Collect coverage from cc_binary data deps of java_test
   + 3442179:
     Configure Apple crosstool to return a complete target triple
     that includes minimum OS version and target environment
   + bb6f1a7:
     Collect C++ lcov coverage if runtime object not in runfiles
   + dbb6e99:
     Fixing dependencies of //external package
   + f0213bb:
     [5.2] Upgrade Google Auth Version (#15383)
   + a1a74c9:
     Fix chocolatey package - docsUrl must not 404 (#15395)
   + fe644be:
     Fix cache leak when applying transitions when only a rule's
     attributes change.
   + ad74d52:
     Fix checking remote cache for omitted files in buildevent file
     (#15405)
   + ac21910:
     fix(bzlmod): throw on json parse exception
   + 3d85b88:
     Add a flag to expose undeclared test outputs in unzipped form.
     (#15431)
   + abd7a9f:
     Remove -U_FORTIFY_SOURCE when thin_lto is enabled (#15433)
   + 53b9cb8:
     Catch NumberFormatException while trying to parse thread id.
   + 19740b5:
     Improve the --sandbox_debug error message
   + 0a2a43f:
     Set keywords on appropriate lifecycle events.
   + 394ddb8:
     Record additional profiling information for remotely executed
     actions.
   + 652b48e:
     Fix downloading remote execution output files inside output
     dirs. (#15444)
   + 73f1ecb:
     Fix android emulator darwin_arm64 select
   + 2649c7c:
     Fix --use_top_level_targets_for_symlinks with aliases (#15446)
   + fa1081c:
     Filter libtool warning about table of contents
   + 26f8783:
     Unify sandbox/remote handling of empty TreeArtifact inputs
     (#15449)
   + 6b21b77:
     Revert "Fixes incorrect install names on darwin platforms"
   + e133e66:
     config doesn't error on duplicate `--define` values (#15473)
   + 84d5917:
     Collect coverage from cc_binary data deps of py_test (#15298)
   + 519d2da:
     SolibSymlinkAction does not need exec platform or properties
   + 6e54699:
     Let Starlark tests inherit env variables (#15217)
   + 9610ae8:
     Update PythonZipper action to use CommandLineItem.CapturingMapFn
   + 2f1ff6f:
     Make `coverage --combined_report=lcov` skip incompatible tests
   + 9fad5a3:
     Disable ReturnValueIgnored checks to unblock java_tools release
   + 0120118:
     Bump the limit of Bazel install base size (#15585)
   + 668805a:
     Upgrade zlib to 1.2.12
   + 4d900ce:
     [5.2] Remote: Fix a bug that outputs of actions tagged with
     no-remote are u... (#15453)
   + b703cb9:
     Add feature to produce serialized diagnostics files (#15600)
   + 2e8458b:
     Release 5.2.0 (2022-06-07)
   + 536f8d9:
     Fix fail message construction in cc_shared_library
   + 2d42925:
     Define cc-compiler-darwin in Xcode toolchain
   + a1d7d1f:
     Fix alwayslink in objc_import
   + d273cb6:
     Unify URL/URLs parameter code across http_archive, http_file,
     http_jar
   + fea32be:
     Preserve --experimental_allow_unresolved_symlinks in exec cfg
   + e4bc370:
     Ck/cherry pick cc shared library (#15754)
   + dbdfa07:
     Let Starlark executable rules specify their environment (#15766)
   + e2a6a2b:
     Fix string formatting when java_home path is missing.
   + d54a288:
     Optionally enable LLVM profile continuous mode
   + ad17b44:
     Print remote execution message when the action times out (#15772)
   + 240e3d1:
     Add missing line to cherrypick
     e4bc370 (#15784)
   + 804b474:
     Replace strdupa with strdup
   + 62be9ea:
     Bzlmod: Better canonical repo names for modules with overrides
     (#15793)
   + d4663a1:
     Add repo env test (#15768)
   + 594962c:
     Add is_root struct field to bazel_module (#15815)
   + 3dd2b93:
     Fix null pointer crash with `bazel coverage` on only
     incompatible tests
   + 4175018:
     Add util for finding credential helper to use
   + 3ea9eb2:
     Merge ManifestMergerAction-related commits into release-5.3.0
     (#15824)
   + 64571a4:
     Ck/cherrypick 15669 (#15788)
   + 1404651:
     Create output directories for remote execution (#15818)
   + ae523f8:
     Use tree artifacts in bootclasspath rule
   + 37f181c:
     [credentialhelper] Add types to communicate with the subprocess
   + 06ca634:
     Add a flag to force Bazel to download certain artifacts when
     using --remote_download_minimal (#15870)
   + d35f923:
     RemoteExecutionService: fix outputs not being uploaded
   + 78af34f:
     Cherry-pick proto_lang_toolchain Starlarkfication and
     proto_common module (#15854)
   + afb434d:
     Fix behavior of `print()` in module extensions
   + 6714c30:
     [credentialhelper] Implement invoking credential helper as
     subprocess
   + 0f05904:
     Add register_{execution_platforms,toolchains} directives to
     MODULE.bazel files (#15852)
   + 33516e2:
     [remote] Improve .netrc test in RemoteModuleTest
   + aa2a1f3:
     Fix ZipDecompressor windows 0x80 (file attribute normal)
   + 30f16e5:
     Replace uses of `cfg = "host"` with `cfg = "exec"` (#15922)
   + 2a8d0ad:
     target pattern file: allow comments
   + 6f73205:
     Add factory for creating paths relative to well-known roots
     (#15931)
   + 32cc8e6:
     Update CODEOWNERS (#15910)
   + 63bc14b:
     Implement native analysis_test call. (#15940)
   + 4df77f7:
     Increase osx_cc_configure timeouts
   + cdf01a3:
     Allow string_list flags to be set via repeated flag uses
   + 05e758d:
     [credentialhelper] Add parser for flag syntax (#15929)
   + e4ee344:
     Docs should mention the new no-remote-cache-upload tag (#15965)
   + 96d23d3:
     Add netrc support to --bes_backend (#15970)
   + c5bc34e:
     Add CommandLinePathFactory to CommandEnvironment (#15971)
   + 508f185:
     Move newCredentialHelperProvider into GoogleAuthUtils (#15973)
   + 14c944a:
     Wire up credential helper to command-line flag(s) (#15976)
   + 04c373b:
     Add `--output=files` mode to cquery (#15979)
   + edfe2a1:
     Make cpp assembly file extensions case sensitive again
   + 4ae8538:
     Prevent aspects from executing on incompatible targets (#15984)
   + f440f8e:
     Remote: Fix performance regression in "upload missing inputs".
     (#15998)
   + 0109031:
     Updated Codeowners file (#16032)
   + 6102d33:
     Propagate the error message when a credential helper fails.
     (#16030)
   + a8dacc7:
     Migrate legacy desugar wrapper to new rlocation() (#16025)
   + 11368be:
     Correctly report errors thrown by CommandLinePathFactory#create.
   + 82452c7:
     Fix an issue that
     `incompatible_remote_build_event_upload_respect_no_… (#16045)
   + e745468:
     Fix rpath for binaries in external repositories (#16079)
   + 83041b1:
     Refactor combined cache. (#16110)
   + c62496f:
     C++: Add compound error linked statically but not exported
     (#16113)
   + 0f18786:
     Do not crash on URIs without a host component.
   + 9c0940d:
     Add profiler task for calling a credential helper.
   + 2ca1ab2:
     Make bazel_cc_code_coverage_test more robust against GCC version
     differences (#16254)
   + 1e25152:
     Fix local execution of external dynamically linked cc_* targets
     (#16253)
   + f6cccae:
     * add change to allow blaze info to skip Starlark build settings
     that start with --no prefix * add unit tests for both info and
     clean commands
   + 59b8b8f:
     Release 5.3.1 (2022-09-19)
   + 77f0233:
     Update GrpcRemoteDownloader to only include relevant headers.
     (#16450)
   + 42ff95a:
     Avoid unnecessary iteration on action inputs.
   + d29034e:
     Update flag `--experimental_remote_download_regex` to accept
     multiple regular expressions. (#16478)
   + bc087f4:
     Release 5.3.2 (2022-10-19)
   + 0b914c6:
     Send remote actions to specific worker pools instead of machine
     types.
   + ece17d5:
     Add `$(rlocationpath(s) ...)` expansion (#16668)
   + f02bcf8:
     Fix identical gcov json file name problem
   + 0696b8a:
     Upgrade google-http-client and google-http-client-gson.
   + 42a3dbb:
     Move analysis_test into testing.analysis_test (#16702)
   + b55f322:
     Fix hanging issue when Bazel failed to upload action inputs
     (#16819)
   + 2f0f3e1:
     [5.4.0] Add integration tests for
     --experimental_credential_helper. (#16880)
   + 6d2d68d:
     [5.4.0] Keep credentials cached across build commands. (#16884)
   + 676a0c8:
     Update Bazel to depend on bazelbuild/platforms 0.0.5.
   + 0ea070b:
     Backport recent package metadata and license check capabilities
     from Bazel 6.x. (#16892)
   + b51396a:
     Add 'toolchain' parameter to actions.{run,run_shell} (#16964)
   + 312fcab:
     Release 5.4.0 (2022-12-15)
   + 0c4e292:
     Pin Bazel version to 5.4.0 (#17986)
   + 43dadb2:
     Bump minimum supported macOS versions to 10.13
   + 1f2b3ed:
     Patch zlib to fix compatibility with latest Xcode
   + a35f592:
     Use ctime in file digest cache key (#18115)
   + e6af231:
     [5.4.1] Disable failing tests (#18123)

Incompatible changes:

  - GrpcRemoteDownloader only includes relevant headers instead of
    sending all credentials.

    Closes #16439.
  - analysis_test moved into testing.analysis_test

Important changes:

  - alias() can now select() directly on constraint_value()

    Fixes #13047.

    Closes #14310.
  - Fixed an issue where Bazel could erroneously report a test passes
    in coverage mode without actually running the test.
  - Make protocOpts() publicly accessible.
  - Add coverage configuration fragment, used to expose
    output_generator label.
  - Bazel now no longer includes system headers on macOS in coverage
    reports (#14969).

    Closes #14971.
  - Starlark test rules can use the new inherited_environment
    parameter of testing.TestEnvironment to specify environment
    variables
    whose values should be inherited from the shell environment.

    Closes #14849.
  - none
    RELNOTES:none
  - Enable merging permissions during Android manifest merging with
    the --merge_android_manifest_permissions flag.
  - Added new register_{execution_platforms,toolchains} directives to
    the MODULE.bazel file, to replace the
    {execution_platforms,toolchains}_to_register attributes on the
    module() directive.
  - Add support for fetching RPC credentials from credential helper.

    Progress on #15856

    Closes #15947.
  - `cquery`'s new output mode
    [`--output=files`](https://bazel.build/docs/cquery#files-output)
    lists the output files of the targets matching the query. It
    takes the current value of `--output_groups` into account.

    Closes #15552.
  - Fix for desugaring failure on Bazel+Android+Windows build
    scenario.
  - The new path variable `$(rlocationpath ...)` and its plural form
    `$(rlocationpaths ...)` can be used to expand labels to the paths
    accepted by the `Rlocation` function of runfiles libraries. This
    is the preferred way to access data dependencies at runtime and
    works on all platforms, even when runfiles are not enabled (e.g.,
    on Windows by default).

    Work towards #16124
    Fixes #10923

    Closes #16667.

This release contains contributions from many people at Google, as well as amberdixon, Andreas Fuchs, Benjamin Peterson, Brentley Jones, Chenchu Kolli, Dan Fleming, Danny Wolf, Emil Kattainen, Fabian Meumertzheim, Gowroji Sunil, hvadehra, Juh-Roch, Keith Smiley, Krzysztof Naglik, kshyanashree, Niyas Sait, Noa Resare, Oliver Eikemeier, oquenchil, Peter Mounce, Philipp Schrader, Ryan Beasley, Thi Doãn, Xùdōng Yáng, Yannic, Zhongpeng Lin.
apattidb pushed a commit to databricks/bazel that referenced this pull request Apr 24, 2023
@keertk
Release 5.4.1 (2023-04-19)
757c24e 
Baseline: 8d66a41

Cherry picks:

   + becd149:
     Remote: Cache merkle trees
   + d7628e1:
     Update DEFAULT_IOS_CPU for M1 arm64 simulator support
   + 80c56ff:
     Compile Apple tools as fat binaries if possible
   + 3c09f34:
     Add protobuf as a well known module
   + 3a5b360:
     Remote: Merge target-level exec_properties with
     --remote_default_exec_properties
   + 917e15e:
     Add -no_uuid for hermetic macOS toolchain setup
   + f5cf8b0:
     Remote: Fixes an issue when --experimental_remote_cache_async
     encounter flaky tests.
   + 77a002c:
     Remove DigestUtils.getDigestInExclusiveMode() now that SsdModule
     has …
   + 557a7e7:
     Fixes for the Starlark transition hash computation (bazelbuild#14251)
   + 34c7146:
     Do location expansion in copts of objc_library
   + 50274a9:
     [5.x] Remote: Add support for compression on gRPC cache (bazelbuild#14277)
   + 61bf2e5:
     Automated rollback of commit
     34c7146.
   + 79888fe:
     Silence a zstd-jni GCC warning.
   + 063b5c9:
     Remote: Limit max number of gRPC connections by
     --remote_max_connections.
   + fd727ec:
     Do location expansion in copts of objc_library
   + 23d0969:
     Fix _is_shared_library_extension_valid
   + 5cf1d6e:
     Remove merging of java_outputs in JavaPluginInfo.
   + cea5f4f:
     Cherrypick Bzlmod documentation (bazelbuild#14301)
   + 227e49e:
     Format work requests according to ndjson spec
   + ae0a6c9:
     Enable user_link_flags_feature for macosx cc_toolchain_config
   + 8c2c78c:
     Remote: Use Action's salt field to differentiate cache across
     workspaces.
   + f948989:
     [5.x] Remote: Fix "file not found" error when remote cache is
     changed from enabled to disabled.  (bazelbuild#14321)
   + 3069ac4:
     Delete marker file before fetching an external repository
   + c05c626:
     Remote: Fix file counting in merkletree.DirectoryTreeBuilder
   + d84f799:
     Fix remote spawn tests for remote_merkle_tree_cache=true
   + 59e16e9:
     Show skipped tests as a warning
   + 76b3c24:
     Build xcode-locator as a universal binary
   + aa52f2d:
     Exit collect_coverage.sh early if LCOV_MERGER is not set.
   + 4256d46:
     Automated rollback of commit
     d84f799.
   + dce2435:
     [apple] fix issues compiling C in objc_library for watchos/armv7k
   + bfc2413:
     5.x: Remote: Ignore blobs referenced in BEP if the generating
     action cannot be cached remotely. (bazelbuild#14389)
   + 5aef53a:
     Remote: Don't blocking-get when acquiring gRPC connections.
     (bazelbuild#14420)
   + 005361c:
     Disable IncludeValidation for ObjC in bazel
   + d703b7b:
     Update java_tools v11.6
   + 90965b0:
     Stop remote blob upload if upload is complete. (bazelbuild#14467)
   + dc59d9e:
     [5.x] Make remote BES uploader better (bazelbuild#14472)
   + 2edab73:
     Avoid too verbose warnings in terminal when cache issues
   + 1160485:
     Rename --project_id to --bes_instance_name
   + c63d9ec:
     Automated rollback of commit
     bfdfa6e.
   + b341802:
     [apple] support watchos_arm64 in toolchain
   + 43bcf80:
     Disable implicitly collecting baseline coverage for toolchain
     targets.
   + 302971e:
     Automated rollback of commit
     7d09b4a.
   + 6200202:
     Bzlmod: Starlarkify default attr values for TypeCheckedTags
   + 38117d4:
     Fix build after rc4 cherrypicks (bazelbuild#14581)
   + 41feb61:
     Release 5.0.0 (2022-01-19)
   + 486d153:
     Find runfiles in directories that are themselves runfiles
   + 0de7bb9:
     Don't resolve symlinks for --sandbox_base
   + 8b60c90:
     Remove uses of -lstdc++ on darwin
   + 60f757c:
     Allow Label instances as keys in select (bazelbuild#14755)
   + 3836ad0:
     Remote: Only waits for background tasks from remote execution.
   + 8734ccf:
     Add the default solib dir to the rpath for cc_imports with
     transitions
   + 9e16a64:
     Flip --experimental_worker_allow_json_protocol
   + fce7ea8:
     Fix `ctx.fragments.apple.single_arch_cpu` returning incorrect
     cpu for tools when host cpu and exec cpu are different
   + 0c1d09e:
     Propagate --experimental_cc_implementation_deps to host config
   + 1c3a245:
     Support select() on constraint_value for aliases.
   + 67a133b:
     Improve documentation for select()
   + 5356fed:
     Cherrypicks for experimental cc_shared_library (bazelbuild#14773)
   + ffdd633:
     [apple] support tvos_sim_arm64 in toolchain (bazelbuild#14779)
   + a58ddea:
     Cherry pick win arm64 (bazelbuild#14794)
   + dc41a20:
     [5.1.0] cherrypick subpackages support (bazelbuild#14780)
   + 86e2db7:
     Add a helper method for rules to depend on the cpp toolchain
     type.
   + 6990c02:
     UrlRewriter should be able to load credentials from .netrc
     (bazelbuild#14834)
   + 32d1606:
     Add "arch" struct field to repository_os
   + 2cfdcea:
     [5.x] bzlmod: Add support for WORKSPACE.bzlmod (bazelbuild#14813)
   + c2ddbd1:
     Ignore missing include directory in JDK distribution.
   + 16de035:
     Fix bazel coverage false negative
   + 0c74741:
     Remote: Postpone the block waiting in `afterCommand` to
     `BlockWaitingModule` (bazelbuild#14833)
   + 3297d92:
     Switch to `ProcessHandle` for getting the PID (bazelbuild#14842)
   + a987b98:
     Fix uses of std++ on bsd
   + d184e48:
     Remote: handle early return of compressed blobs uploads
   + 0b09e9e:
     Add removeprefix/removesuffix to Starlark strings
   + d42ab0c:
     Fix default CPU for macOS and iOS (bazelbuild#14923)
   + cd24f39:
     Add paramfile support for def_parser, since in rare cases on
     Windows command line character limit was reached.
   + 0b1beef:
     Normalize rpath entries to guard against missing default solib
     dir
   + 24e8242:
     Fix aggressive params file assumption
   + c45838b:
     Fix precompiled libs not in runfiles of cc_shared_library
     (bazelbuild#14943)
   + 764614e:
     Bzlmod: Allow multiple `use_extension`s on the same extension
     (bazelbuild#14945)
   + fa761f8:
     Fix typo in `apple_common.platform` docs
   + f7d8288:
     Yield a Proxy for addresses without protocol
   + 8cefb8b:
     Avoid merging URLs in HttpUtils
   + b480480:
     Make protocOpts() public. (bazelbuild#14952)
   + 113eaca:
     Do not hide BulkTransferException messages when there were more
     than one exception
   + b1bf9d6:
     merkle_tree_cache: change default size to 1000
   + f15e0c7:
     Add --experimental_repository_cache_urls_as_default_canonical_id
     to help detect broken repository URLs (bazelbuild#14989)
   + f421474:
     Expose the logic to read user netrc file
   + b858ec3:
     Correct cpu and os values of `local_config_cc_toolchains` targets
   + 5e79972:
     Expose CoverageOutputGenerator on a Fragment (bazelbuild#14997)
   + 78f0311:
     Correct error runfiles cc_shared_library (bazelbuild#14998)
   + 7937dd1:
     [5.1] Adding Starlark dependencies to the package //external
     (bazelbuild#14991)
   + a73aa12:
     Remote: Fix crashes with InterruptedException when using http
     cache.
   + f8707c0:
     Account for interface libraries in cc_shared_library
   + a570f5f:
     Fix coverage runfiles directory issue
   + 95de355:
     Do not validate input-only settings in transitions (bazelbuild#15048)
   + 71747cc:
     Filter out system headers on macOS.
   + cb6500a:
     Update Bazel bootstrap documentation and remove obsolete flags.
     (bazelbuild#15065)
   + 4c031d1:
     [5.1] Undocument --bes_best_effort (bazelbuild#15066)
   + 267142f:
     Fix conflicting actions error when specifying
     --host_macos_minimum_os (bazelbuild#15068)
   + f192362:
     [5.1] Remote: Action should not be successful and cached if
     outputs were not created (bazelbuild#15071)
   + 00d74ff:
     Support decompressing zstd tar archives for repository rules.
   + f585783:
     Remote: Don't check TreeArtifact output
   + efb2b80:
     osx_cc_wrapper: Only expand existing response files
   + c771c43:
     Remote: Fix crashes by InterruptedException when dynamic
     execution is enabled. (bazelbuild#15091)
   + 3785677:
     Use python3 on macOS
   + 815d9e4:
     Release 5.1.0 (2022-03-24)
   + 1fbb69e:
     Prepare 5.1.1 release
   + df153df:
     Fix CODEOWNERS syntax
   + 2b92a31:
     Remote: Don't check declared outputs for failed action
   + b47aa71:
     Upgrade abseil version to the latest
   + c49c45d:
     Revert default export all symbols on Windows
   + 7d3fb99:
     Support ZIP files with total number of disks = 0
   + 0f5dc11:
     Release 5.1.1 (2022-04-08)
   + 2422cfb:
     Update CODEOWNERS
   + bbcff18:
     [5.2.0] Update java_tools 11.7.1 (bazelbuild#15231)
   + 9c98120:
     Add support for .ar archives (and .deb files)
   + d3435b0:
     Seperate GetSelfPath implementation for Blaze and Bazel
   + c94572b:
     Include jdk.crypto.mscapi in minimized Windows embedded JDK
   + 299022c:
     remote: Proactively close the ZstdInputStream in
     ZstdDecompressingOutputStream.
   + 2770799:
     Collect coverage from cc_binary data deps of java_test
   + 3442179:
     Configure Apple crosstool to return a complete target triple
     that includes minimum OS version and target environment
   + bb6f1a7:
     Collect C++ lcov coverage if runtime object not in runfiles
   + dbb6e99:
     Fixing dependencies of //external package
   + f0213bb:
     [5.2] Upgrade Google Auth Version (bazelbuild#15383)
   + a1a74c9:
     Fix chocolatey package - docsUrl must not 404 (bazelbuild#15395)
   + fe644be:
     Fix cache leak when applying transitions when only a rule's
     attributes change.
   + ad74d52:
     Fix checking remote cache for omitted files in buildevent file
     (bazelbuild#15405)
   + ac21910:
     fix(bzlmod): throw on json parse exception
   + 3d85b88:
     Add a flag to expose undeclared test outputs in unzipped form.
     (bazelbuild#15431)
   + abd7a9f:
     Remove -U_FORTIFY_SOURCE when thin_lto is enabled (bazelbuild#15433)
   + 53b9cb8:
     Catch NumberFormatException while trying to parse thread id.
   + 19740b5:
     Improve the --sandbox_debug error message
   + 0a2a43f:
     Set keywords on appropriate lifecycle events.
   + 394ddb8:
     Record additional profiling information for remotely executed
     actions.
   + 652b48e:
     Fix downloading remote execution output files inside output
     dirs. (bazelbuild#15444)
   + 73f1ecb:
     Fix android emulator darwin_arm64 select
   + 2649c7c:
     Fix --use_top_level_targets_for_symlinks with aliases (bazelbuild#15446)
   + fa1081c:
     Filter libtool warning about table of contents
   + 26f8783:
     Unify sandbox/remote handling of empty TreeArtifact inputs
     (bazelbuild#15449)
   + 6b21b77:
     Revert "Fixes incorrect install names on darwin platforms"
   + e133e66:
     config doesn't error on duplicate `--define` values (bazelbuild#15473)
   + 84d5917:
     Collect coverage from cc_binary data deps of py_test (bazelbuild#15298)
   + 519d2da:
     SolibSymlinkAction does not need exec platform or properties
   + 6e54699:
     Let Starlark tests inherit env variables (bazelbuild#15217)
   + 9610ae8:
     Update PythonZipper action to use CommandLineItem.CapturingMapFn
   + 2f1ff6f:
     Make `coverage --combined_report=lcov` skip incompatible tests
   + 9fad5a3:
     Disable ReturnValueIgnored checks to unblock java_tools release
   + 0120118:
     Bump the limit of Bazel install base size (bazelbuild#15585)
   + 668805a:
     Upgrade zlib to 1.2.12
   + 4d900ce:
     [5.2] Remote: Fix a bug that outputs of actions tagged with
     no-remote are u... (bazelbuild#15453)
   + b703cb9:
     Add feature to produce serialized diagnostics files (bazelbuild#15600)
   + 2e8458b:
     Release 5.2.0 (2022-06-07)
   + 536f8d9:
     Fix fail message construction in cc_shared_library
   + 2d42925:
     Define cc-compiler-darwin in Xcode toolchain
   + a1d7d1f:
     Fix alwayslink in objc_import
   + d273cb6:
     Unify URL/URLs parameter code across http_archive, http_file,
     http_jar
   + fea32be:
     Preserve --experimental_allow_unresolved_symlinks in exec cfg
   + e4bc370:
     Ck/cherry pick cc shared library (bazelbuild#15754)
   + dbdfa07:
     Let Starlark executable rules specify their environment (bazelbuild#15766)
   + e2a6a2b:
     Fix string formatting when java_home path is missing.
   + d54a288:
     Optionally enable LLVM profile continuous mode
   + ad17b44:
     Print remote execution message when the action times out (bazelbuild#15772)
   + 240e3d1:
     Add missing line to cherrypick
     e4bc370 (bazelbuild#15784)
   + 804b474:
     Replace strdupa with strdup
   + 62be9ea:
     Bzlmod: Better canonical repo names for modules with overrides
     (bazelbuild#15793)
   + d4663a1:
     Add repo env test (bazelbuild#15768)
   + 594962c:
     Add is_root struct field to bazel_module (bazelbuild#15815)
   + 3dd2b93:
     Fix null pointer crash with `bazel coverage` on only
     incompatible tests
   + 4175018:
     Add util for finding credential helper to use
   + 3ea9eb2:
     Merge ManifestMergerAction-related commits into release-5.3.0
     (bazelbuild#15824)
   + 64571a4:
     Ck/cherrypick 15669 (bazelbuild#15788)
   + 1404651:
     Create output directories for remote execution (bazelbuild#15818)
   + ae523f8:
     Use tree artifacts in bootclasspath rule
   + 37f181c:
     [credentialhelper] Add types to communicate with the subprocess
   + 06ca634:
     Add a flag to force Bazel to download certain artifacts when
     using --remote_download_minimal (bazelbuild#15870)
   + d35f923:
     RemoteExecutionService: fix outputs not being uploaded
   + 78af34f:
     Cherry-pick proto_lang_toolchain Starlarkfication and
     proto_common module (bazelbuild#15854)
   + afb434d:
     Fix behavior of `print()` in module extensions
   + 6714c30:
     [credentialhelper] Implement invoking credential helper as
     subprocess
   + 0f05904:
     Add register_{execution_platforms,toolchains} directives to
     MODULE.bazel files (bazelbuild#15852)
   + 33516e2:
     [remote] Improve .netrc test in RemoteModuleTest
   + aa2a1f3:
     Fix ZipDecompressor windows 0x80 (file attribute normal)
   + 30f16e5:
     Replace uses of `cfg = "host"` with `cfg = "exec"` (bazelbuild#15922)
   + 2a8d0ad:
     target pattern file: allow comments
   + 6f73205:
     Add factory for creating paths relative to well-known roots
     (bazelbuild#15931)
   + 32cc8e6:
     Update CODEOWNERS (bazelbuild#15910)
   + 63bc14b:
     Implement native analysis_test call. (bazelbuild#15940)
   + 4df77f7:
     Increase osx_cc_configure timeouts
   + cdf01a3:
     Allow string_list flags to be set via repeated flag uses
   + 05e758d:
     [credentialhelper] Add parser for flag syntax (bazelbuild#15929)
   + e4ee344:
     Docs should mention the new no-remote-cache-upload tag (bazelbuild#15965)
   + 96d23d3:
     Add netrc support to --bes_backend (bazelbuild#15970)
   + c5bc34e:
     Add CommandLinePathFactory to CommandEnvironment (bazelbuild#15971)
   + 508f185:
     Move newCredentialHelperProvider into GoogleAuthUtils (bazelbuild#15973)
   + 14c944a:
     Wire up credential helper to command-line flag(s) (bazelbuild#15976)
   + 04c373b:
     Add `--output=files` mode to cquery (bazelbuild#15979)
   + edfe2a1:
     Make cpp assembly file extensions case sensitive again
   + 4ae8538:
     Prevent aspects from executing on incompatible targets (bazelbuild#15984)
   + f440f8e:
     Remote: Fix performance regression in "upload missing inputs".
     (bazelbuild#15998)
   + 0109031:
     Updated Codeowners file (bazelbuild#16032)
   + 6102d33:
     Propagate the error message when a credential helper fails.
     (bazelbuild#16030)
   + a8dacc7:
     Migrate legacy desugar wrapper to new rlocation() (bazelbuild#16025)
   + 11368be:
     Correctly report errors thrown by CommandLinePathFactory#create.
   + 82452c7:
     Fix an issue that
     `incompatible_remote_build_event_upload_respect_no_… (bazelbuild#16045)
   + e745468:
     Fix rpath for binaries in external repositories (bazelbuild#16079)
   + 83041b1:
     Refactor combined cache. (bazelbuild#16110)
   + c62496f:
     C++: Add compound error linked statically but not exported
     (bazelbuild#16113)
   + 0f18786:
     Do not crash on URIs without a host component.
   + 9c0940d:
     Add profiler task for calling a credential helper.
   + 2ca1ab2:
     Make bazel_cc_code_coverage_test more robust against GCC version
     differences (bazelbuild#16254)
   + 1e25152:
     Fix local execution of external dynamically linked cc_* targets
     (bazelbuild#16253)
   + f6cccae:
     * add change to allow blaze info to skip Starlark build settings
     that start with --no prefix * add unit tests for both info and
     clean commands
   + 59b8b8f:
     Release 5.3.1 (2022-09-19)
   + 77f0233:
     Update GrpcRemoteDownloader to only include relevant headers.
     (bazelbuild#16450)
   + 42ff95a:
     Avoid unnecessary iteration on action inputs.
   + d29034e:
     Update flag `--experimental_remote_download_regex` to accept
     multiple regular expressions. (bazelbuild#16478)
   + bc087f4:
     Release 5.3.2 (2022-10-19)
   + 0b914c6:
     Send remote actions to specific worker pools instead of machine
     types.
   + ece17d5:
     Add `$(rlocationpath(s) ...)` expansion (bazelbuild#16668)
   + f02bcf8:
     Fix identical gcov json file name problem
   + 0696b8a:
     Upgrade google-http-client and google-http-client-gson.
   + 42a3dbb:
     Move analysis_test into testing.analysis_test (bazelbuild#16702)
   + b55f322:
     Fix hanging issue when Bazel failed to upload action inputs
     (bazelbuild#16819)
   + 2f0f3e1:
     [5.4.0] Add integration tests for
     --experimental_credential_helper. (bazelbuild#16880)
   + 6d2d68d:
     [5.4.0] Keep credentials cached across build commands. (bazelbuild#16884)
   + 676a0c8:
     Update Bazel to depend on bazelbuild/platforms 0.0.5.
   + 0ea070b:
     Backport recent package metadata and license check capabilities
     from Bazel 6.x. (bazelbuild#16892)
   + b51396a:
     Add 'toolchain' parameter to actions.{run,run_shell} (bazelbuild#16964)
   + 312fcab:
     Release 5.4.0 (2022-12-15)
   + 0c4e292:
     Pin Bazel version to 5.4.0 (bazelbuild#17986)
   + 43dadb2:
     Bump minimum supported macOS versions to 10.13
   + 1f2b3ed:
     Patch zlib to fix compatibility with latest Xcode
   + a35f592:
     Use ctime in file digest cache key (bazelbuild#18115)
   + e6af231:
     [5.4.1] Disable failing tests (bazelbuild#18123)

Incompatible changes:

  - GrpcRemoteDownloader only includes relevant headers instead of
    sending all credentials.

    Closes bazelbuild#16439.
  - analysis_test moved into testing.analysis_test

Important changes:

  - alias() can now select() directly on constraint_value()

    Fixes bazelbuild#13047.

    Closes bazelbuild#14310.
  - Fixed an issue where Bazel could erroneously report a test passes
    in coverage mode without actually running the test.
  - Make protocOpts() publicly accessible.
  - Add coverage configuration fragment, used to expose
    output_generator label.
  - Bazel now no longer includes system headers on macOS in coverage
    reports (bazelbuild#14969).

    Closes bazelbuild#14971.
  - Starlark test rules can use the new inherited_environment
    parameter of testing.TestEnvironment to specify environment
    variables
    whose values should be inherited from the shell environment.

    Closes bazelbuild#14849.
  - none
    RELNOTES:none
  - Enable merging permissions during Android manifest merging with
    the --merge_android_manifest_permissions flag.
  - Added new register_{execution_platforms,toolchains} directives to
    the MODULE.bazel file, to replace the
    {execution_platforms,toolchains}_to_register attributes on the
    module() directive.
  - Add support for fetching RPC credentials from credential helper.

    Progress on bazelbuild#15856

    Closes bazelbuild#15947.
  - `cquery`'s new output mode
    [`--output=files`](https://bazel.build/docs/cquery#files-output)
    lists the output files of the targets matching the query. It
    takes the current value of `--output_groups` into account.

    Closes bazelbuild#15552.
  - Fix for desugaring failure on Bazel+Android+Windows build
    scenario.
  - The new path variable `$(rlocationpath ...)` and its plural form
    `$(rlocationpaths ...)` can be used to expand labels to the paths
    accepted by the `Rlocation` function of runfiles libraries. This
    is the preferred way to access data dependencies at runtime and
    works on all platforms, even when runfiles are not enabled (e.g.,
    on Windows by default).

    Work towards bazelbuild#16124
    Fixes bazelbuild#10923

    Closes bazelbuild#16667.

This release contains contributions from many people at Google, as well as amberdixon, Andreas Fuchs, Benjamin Peterson, Brentley Jones, Chenchu Kolli, Dan Fleming, Danny Wolf, Emil Kattainen, Fabian Meumertzheim, Gowroji Sunil, hvadehra, Juh-Roch, Keith Smiley, Krzysztof Naglik, kshyanashree, Niyas Sait, Noa Resare, Oliver Eikemeier, oquenchil, Peter Mounce, Philipp Schrader, Ryan Beasley, Thi Doãn, Xùdōng Yáng, Yannic, Zhongpeng Lin.
@lijunsong
Copy link

lijunsong commented May 2, 2023 •
edited
Loading

I want to point out that this release seems to be small , but its impact is larger than expected. Since the action key hash is changed, Bazel 5.4.1 doesn't use the existing remote cache anymore. The impact is sudden increases of build time (due to a drop of cache hit rate) and a potentially large traffic to remote build service as well.

@fmeum
Copy link
Collaborator Author

fmeum commented May 2, 2023

@lijunsong Could you describe the exact effect you are seeing in more detail?

This PR should only affect the file digest cache, which is not persisted across server restarts anyway - it's effect thus shouldn't be worse than running bazel shutdown, which anyone upgrading to 5.4.1 implicitly did anyway.

If it ends up affecting any kind of persistent cache, then there is definitely a regression that we should understand and fix.

fweikert pushed a commit to fweikert/bazel that referenced this pull request May 25, 2023
@fweikert
Release 5.4.1 (2023-04-19)
25c0679 
Baseline: 8d66a41

Cherry picks:

   + becd149:
     Remote: Cache merkle trees
   + d7628e1:
     Update DEFAULT_IOS_CPU for M1 arm64 simulator support
   + 80c56ff:
     Compile Apple tools as fat binaries if possible
   + 3c09f34:
     Add protobuf as a well known module
   + 3a5b360:
     Remote: Merge target-level exec_properties with
     --remote_default_exec_properties
   + 917e15e:
     Add -no_uuid for hermetic macOS toolchain setup
   + f5cf8b0:
     Remote: Fixes an issue when --experimental_remote_cache_async
     encounter flaky tests.
   + 77a002c:
     Remove DigestUtils.getDigestInExclusiveMode() now that SsdModule
     has …
   + 557a7e7:
     Fixes for the Starlark transition hash computation (bazelbuild#14251)
   + 34c7146:
     Do location expansion in copts of objc_library
   + 50274a9:
     [5.x] Remote: Add support for compression on gRPC cache (bazelbuild#14277)
   + 61bf2e5:
     Automated rollback of commit
     34c7146.
   + 79888fe:
     Silence a zstd-jni GCC warning.
   + 063b5c9:
     Remote: Limit max number of gRPC connections by
     --remote_max_connections.
   + fd727ec:
     Do location expansion in copts of objc_library
   + 23d0969:
     Fix _is_shared_library_extension_valid
   + 5cf1d6e:
     Remove merging of java_outputs in JavaPluginInfo.
   + cea5f4f:
     Cherrypick Bzlmod documentation (bazelbuild#14301)
   + 227e49e:
     Format work requests according to ndjson spec
   + ae0a6c9:
     Enable user_link_flags_feature for macosx cc_toolchain_config
   + 8c2c78c:
     Remote: Use Action's salt field to differentiate cache across
     workspaces.
   + f948989:
     [5.x] Remote: Fix "file not found" error when remote cache is
     changed from enabled to disabled.  (bazelbuild#14321)
   + 3069ac4:
     Delete marker file before fetching an external repository
   + c05c626:
     Remote: Fix file counting in merkletree.DirectoryTreeBuilder
   + d84f799:
     Fix remote spawn tests for remote_merkle_tree_cache=true
   + 59e16e9:
     Show skipped tests as a warning
   + 76b3c24:
     Build xcode-locator as a universal binary
   + aa52f2d:
     Exit collect_coverage.sh early if LCOV_MERGER is not set.
   + 4256d46:
     Automated rollback of commit
     d84f799.
   + dce2435:
     [apple] fix issues compiling C in objc_library for watchos/armv7k
   + bfc2413:
     5.x: Remote: Ignore blobs referenced in BEP if the generating
     action cannot be cached remotely. (bazelbuild#14389)
   + 5aef53a:
     Remote: Don't blocking-get when acquiring gRPC connections.
     (bazelbuild#14420)
   + 005361c:
     Disable IncludeValidation for ObjC in bazel
   + d703b7b:
     Update java_tools v11.6
   + 90965b0:
     Stop remote blob upload if upload is complete. (bazelbuild#14467)
   + dc59d9e:
     [5.x] Make remote BES uploader better (bazelbuild#14472)
   + 2edab73:
     Avoid too verbose warnings in terminal when cache issues
   + 1160485:
     Rename --project_id to --bes_instance_name
   + c63d9ec:
     Automated rollback of commit
     bfdfa6e.
   + b341802:
     [apple] support watchos_arm64 in toolchain
   + 43bcf80:
     Disable implicitly collecting baseline coverage for toolchain
     targets.
   + 302971e:
     Automated rollback of commit
     7d09b4a.
   + 6200202:
     Bzlmod: Starlarkify default attr values for TypeCheckedTags
   + 38117d4:
     Fix build after rc4 cherrypicks (bazelbuild#14581)
   + 41feb61:
     Release 5.0.0 (2022-01-19)
   + 486d153:
     Find runfiles in directories that are themselves runfiles
   + 0de7bb9:
     Don't resolve symlinks for --sandbox_base
   + 8b60c90:
     Remove uses of -lstdc++ on darwin
   + 60f757c:
     Allow Label instances as keys in select (bazelbuild#14755)
   + 3836ad0:
     Remote: Only waits for background tasks from remote execution.
   + 8734ccf:
     Add the default solib dir to the rpath for cc_imports with
     transitions
   + 9e16a64:
     Flip --experimental_worker_allow_json_protocol
   + fce7ea8:
     Fix `ctx.fragments.apple.single_arch_cpu` returning incorrect
     cpu for tools when host cpu and exec cpu are different
   + 0c1d09e:
     Propagate --experimental_cc_implementation_deps to host config
   + 1c3a245:
     Support select() on constraint_value for aliases.
   + 67a133b:
     Improve documentation for select()
   + 5356fed:
     Cherrypicks for experimental cc_shared_library (bazelbuild#14773)
   + ffdd633:
     [apple] support tvos_sim_arm64 in toolchain (bazelbuild#14779)
   + a58ddea:
     Cherry pick win arm64 (bazelbuild#14794)
   + dc41a20:
     [5.1.0] cherrypick subpackages support (bazelbuild#14780)
   + 86e2db7:
     Add a helper method for rules to depend on the cpp toolchain
     type.
   + 6990c02:
     UrlRewriter should be able to load credentials from .netrc
     (bazelbuild#14834)
   + 32d1606:
     Add "arch" struct field to repository_os
   + 2cfdcea:
     [5.x] bzlmod: Add support for WORKSPACE.bzlmod (bazelbuild#14813)
   + c2ddbd1:
     Ignore missing include directory in JDK distribution.
   + 16de035:
     Fix bazel coverage false negative
   + 0c74741:
     Remote: Postpone the block waiting in `afterCommand` to
     `BlockWaitingModule` (bazelbuild#14833)
   + 3297d92:
     Switch to `ProcessHandle` for getting the PID (bazelbuild#14842)
   + a987b98:
     Fix uses of std++ on bsd
   + d184e48:
     Remote: handle early return of compressed blobs uploads
   + 0b09e9e:
     Add removeprefix/removesuffix to Starlark strings
   + d42ab0c:
     Fix default CPU for macOS and iOS (bazelbuild#14923)
   + cd24f39:
     Add paramfile support for def_parser, since in rare cases on
     Windows command line character limit was reached.
   + 0b1beef:
     Normalize rpath entries to guard against missing default solib
     dir
   + 24e8242:
     Fix aggressive params file assumption
   + c45838b:
     Fix precompiled libs not in runfiles of cc_shared_library
     (bazelbuild#14943)
   + 764614e:
     Bzlmod: Allow multiple `use_extension`s on the same extension
     (bazelbuild#14945)
   + fa761f8:
     Fix typo in `apple_common.platform` docs
   + f7d8288:
     Yield a Proxy for addresses without protocol
   + 8cefb8b:
     Avoid merging URLs in HttpUtils
   + b480480:
     Make protocOpts() public. (bazelbuild#14952)
   + 113eaca:
     Do not hide BulkTransferException messages when there were more
     than one exception
   + b1bf9d6:
     merkle_tree_cache: change default size to 1000
   + f15e0c7:
     Add --experimental_repository_cache_urls_as_default_canonical_id
     to help detect broken repository URLs (bazelbuild#14989)
   + f421474:
     Expose the logic to read user netrc file
   + b858ec3:
     Correct cpu and os values of `local_config_cc_toolchains` targets
   + 5e79972:
     Expose CoverageOutputGenerator on a Fragment (bazelbuild#14997)
   + 78f0311:
     Correct error runfiles cc_shared_library (bazelbuild#14998)
   + 7937dd1:
     [5.1] Adding Starlark dependencies to the package //external
     (bazelbuild#14991)
   + a73aa12:
     Remote: Fix crashes with InterruptedException when using http
     cache.
   + f8707c0:
     Account for interface libraries in cc_shared_library
   + a570f5f:
     Fix coverage runfiles directory issue
   + 95de355:
     Do not validate input-only settings in transitions (bazelbuild#15048)
   + 71747cc:
     Filter out system headers on macOS.
   + cb6500a:
     Update Bazel bootstrap documentation and remove obsolete flags.
     (bazelbuild#15065)
   + 4c031d1:
     [5.1] Undocument --bes_best_effort (bazelbuild#15066)
   + 267142f:
     Fix conflicting actions error when specifying
     --host_macos_minimum_os (bazelbuild#15068)
   + f192362:
     [5.1] Remote: Action should not be successful and cached if
     outputs were not created (bazelbuild#15071)
   + 00d74ff:
     Support decompressing zstd tar archives for repository rules.
   + f585783:
     Remote: Don't check TreeArtifact output
   + efb2b80:
     osx_cc_wrapper: Only expand existing response files
   + c771c43:
     Remote: Fix crashes by InterruptedException when dynamic
     execution is enabled. (bazelbuild#15091)
   + 3785677:
     Use python3 on macOS
   + 815d9e4:
     Release 5.1.0 (2022-03-24)
   + 1fbb69e:
     Prepare 5.1.1 release
   + df153df:
     Fix CODEOWNERS syntax
   + 2b92a31:
     Remote: Don't check declared outputs for failed action
   + b47aa71:
     Upgrade abseil version to the latest
   + c49c45d:
     Revert default export all symbols on Windows
   + 7d3fb99:
     Support ZIP files with total number of disks = 0
   + 0f5dc11:
     Release 5.1.1 (2022-04-08)
   + 2422cfb:
     Update CODEOWNERS
   + bbcff18:
     [5.2.0] Update java_tools 11.7.1 (bazelbuild#15231)
   + 9c98120:
     Add support for .ar archives (and .deb files)
   + d3435b0:
     Seperate GetSelfPath implementation for Blaze and Bazel
   + c94572b:
     Include jdk.crypto.mscapi in minimized Windows embedded JDK
   + 299022c:
     remote: Proactively close the ZstdInputStream in
     ZstdDecompressingOutputStream.
   + 2770799:
     Collect coverage from cc_binary data deps of java_test
   + 3442179:
     Configure Apple crosstool to return a complete target triple
     that includes minimum OS version and target environment
   + bb6f1a7:
     Collect C++ lcov coverage if runtime object not in runfiles
   + dbb6e99:
     Fixing dependencies of //external package
   + f0213bb:
     [5.2] Upgrade Google Auth Version (bazelbuild#15383)
   + a1a74c9:
     Fix chocolatey package - docsUrl must not 404 (bazelbuild#15395)
   + fe644be:
     Fix cache leak when applying transitions when only a rule's
     attributes change.
   + ad74d52:
     Fix checking remote cache for omitted files in buildevent file
     (bazelbuild#15405)
   + ac21910:
     fix(bzlmod): throw on json parse exception
   + 3d85b88:
     Add a flag to expose undeclared test outputs in unzipped form.
     (bazelbuild#15431)
   + abd7a9f:
     Remove -U_FORTIFY_SOURCE when thin_lto is enabled (bazelbuild#15433)
   + 53b9cb8:
     Catch NumberFormatException while trying to parse thread id.
   + 19740b5:
     Improve the --sandbox_debug error message
   + 0a2a43f:
     Set keywords on appropriate lifecycle events.
   + 394ddb8:
     Record additional profiling information for remotely executed
     actions.
   + 652b48e:
     Fix downloading remote execution output files inside output
     dirs. (bazelbuild#15444)
   + 73f1ecb:
     Fix android emulator darwin_arm64 select
   + 2649c7c:
     Fix --use_top_level_targets_for_symlinks with aliases (bazelbuild#15446)
   + fa1081c:
     Filter libtool warning about table of contents
   + 26f8783:
     Unify sandbox/remote handling of empty TreeArtifact inputs
     (bazelbuild#15449)
   + 6b21b77:
     Revert "Fixes incorrect install names on darwin platforms"
   + e133e66:
     config doesn't error on duplicate `--define` values (bazelbuild#15473)
   + 84d5917:
     Collect coverage from cc_binary data deps of py_test (bazelbuild#15298)
   + 519d2da:
     SolibSymlinkAction does not need exec platform or properties
   + 6e54699:
     Let Starlark tests inherit env variables (bazelbuild#15217)
   + 9610ae8:
     Update PythonZipper action to use CommandLineItem.CapturingMapFn
   + 2f1ff6f:
     Make `coverage --combined_report=lcov` skip incompatible tests
   + 9fad5a3:
     Disable ReturnValueIgnored checks to unblock java_tools release
   + 0120118:
     Bump the limit of Bazel install base size (bazelbuild#15585)
   + 668805a:
     Upgrade zlib to 1.2.12
   + 4d900ce:
     [5.2] Remote: Fix a bug that outputs of actions tagged with
     no-remote are u... (bazelbuild#15453)
   + b703cb9:
     Add feature to produce serialized diagnostics files (bazelbuild#15600)
   + 2e8458b:
     Release 5.2.0 (2022-06-07)
   + 536f8d9:
     Fix fail message construction in cc_shared_library
   + 2d42925:
     Define cc-compiler-darwin in Xcode toolchain
   + a1d7d1f:
     Fix alwayslink in objc_import
   + d273cb6:
     Unify URL/URLs parameter code across http_archive, http_file,
     http_jar
   + fea32be:
     Preserve --experimental_allow_unresolved_symlinks in exec cfg
   + e4bc370:
     Ck/cherry pick cc shared library (bazelbuild#15754)
   + dbdfa07:
     Let Starlark executable rules specify their environment (bazelbuild#15766)
   + e2a6a2b:
     Fix string formatting when java_home path is missing.
   + d54a288:
     Optionally enable LLVM profile continuous mode
   + ad17b44:
     Print remote execution message when the action times out (bazelbuild#15772)
   + 240e3d1:
     Add missing line to cherrypick
     e4bc370 (bazelbuild#15784)
   + 804b474:
     Replace strdupa with strdup
   + 62be9ea:
     Bzlmod: Better canonical repo names for modules with overrides
     (bazelbuild#15793)
   + d4663a1:
     Add repo env test (bazelbuild#15768)
   + 594962c:
     Add is_root struct field to bazel_module (bazelbuild#15815)
   + 3dd2b93:
     Fix null pointer crash with `bazel coverage` on only
     incompatible tests
   + 4175018:
     Add util for finding credential helper to use
   + 3ea9eb2:
     Merge ManifestMergerAction-related commits into release-5.3.0
     (bazelbuild#15824)
   + 64571a4:
     Ck/cherrypick 15669 (bazelbuild#15788)
   + 1404651:
     Create output directories for remote execution (bazelbuild#15818)
   + ae523f8:
     Use tree artifacts in bootclasspath rule
   + 37f181c:
     [credentialhelper] Add types to communicate with the subprocess
   + 06ca634:
     Add a flag to force Bazel to download certain artifacts when
     using --remote_download_minimal (bazelbuild#15870)
   + d35f923:
     RemoteExecutionService: fix outputs not being uploaded
   + 78af34f:
     Cherry-pick proto_lang_toolchain Starlarkfication and
     proto_common module (bazelbuild#15854)
   + afb434d:
     Fix behavior of `print()` in module extensions
   + 6714c30:
     [credentialhelper] Implement invoking credential helper as
     subprocess
   + 0f05904:
     Add register_{execution_platforms,toolchains} directives to
     MODULE.bazel files (bazelbuild#15852)
   + 33516e2:
     [remote] Improve .netrc test in RemoteModuleTest
   + aa2a1f3:
     Fix ZipDecompressor windows 0x80 (file attribute normal)
   + 30f16e5:
     Replace uses of `cfg = "host"` with `cfg = "exec"` (bazelbuild#15922)
   + 2a8d0ad:
     target pattern file: allow comments
   + 6f73205:
     Add factory for creating paths relative to well-known roots
     (bazelbuild#15931)
   + 32cc8e6:
     Update CODEOWNERS (bazelbuild#15910)
   + 63bc14b:
     Implement native analysis_test call. (bazelbuild#15940)
   + 4df77f7:
     Increase osx_cc_configure timeouts
   + cdf01a3:
     Allow string_list flags to be set via repeated flag uses
   + 05e758d:
     [credentialhelper] Add parser for flag syntax (bazelbuild#15929)
   + e4ee344:
     Docs should mention the new no-remote-cache-upload tag (bazelbuild#15965)
   + 96d23d3:
     Add netrc support to --bes_backend (bazelbuild#15970)
   + c5bc34e:
     Add CommandLinePathFactory to CommandEnvironment (bazelbuild#15971)
   + 508f185:
     Move newCredentialHelperProvider into GoogleAuthUtils (bazelbuild#15973)
   + 14c944a:
     Wire up credential helper to command-line flag(s) (bazelbuild#15976)
   + 04c373b:
     Add `--output=files` mode to cquery (bazelbuild#15979)
   + edfe2a1:
     Make cpp assembly file extensions case sensitive again
   + 4ae8538:
     Prevent aspects from executing on incompatible targets (bazelbuild#15984)
   + f440f8e:
     Remote: Fix performance regression in "upload missing inputs".
     (bazelbuild#15998)
   + 0109031:
     Updated Codeowners file (bazelbuild#16032)
   + 6102d33:
     Propagate the error message when a credential helper fails.
     (bazelbuild#16030)
   + a8dacc7:
     Migrate legacy desugar wrapper to new rlocation() (bazelbuild#16025)
   + 11368be:
     Correctly report errors thrown by CommandLinePathFactory#create.
   + 82452c7:
     Fix an issue that
     `incompatible_remote_build_event_upload_respect_no_… (bazelbuild#16045)
   + e745468:
     Fix rpath for binaries in external repositories (bazelbuild#16079)
   + 83041b1:
     Refactor combined cache. (bazelbuild#16110)
   + c62496f:
     C++: Add compound error linked statically but not exported
     (bazelbuild#16113)
   + 0f18786:
     Do not crash on URIs without a host component.
   + 9c0940d:
     Add profiler task for calling a credential helper.
   + 2ca1ab2:
     Make bazel_cc_code_coverage_test more robust against GCC version
     differences (bazelbuild#16254)
   + 1e25152:
     Fix local execution of external dynamically linked cc_* targets
     (bazelbuild#16253)
   + f6cccae:
     * add change to allow blaze info to skip Starlark build settings
     that start with --no prefix * add unit tests for both info and
     clean commands
   + 59b8b8f:
     Release 5.3.1 (2022-09-19)
   + 77f0233:
     Update GrpcRemoteDownloader to only include relevant headers.
     (bazelbuild#16450)
   + 42ff95a:
     Avoid unnecessary iteration on action inputs.
   + d29034e:
     Update flag `--experimental_remote_download_regex` to accept
     multiple regular expressions. (bazelbuild#16478)
   + bc087f4:
     Release 5.3.2 (2022-10-19)
   + 0b914c6:
     Send remote actions to specific worker pools instead of machine
     types.
   + ece17d5:
     Add `$(rlocationpath(s) ...)` expansion (bazelbuild#16668)
   + f02bcf8:
     Fix identical gcov json file name problem
   + 0696b8a:
     Upgrade google-http-client and google-http-client-gson.
   + 42a3dbb:
     Move analysis_test into testing.analysis_test (bazelbuild#16702)
   + b55f322:
     Fix hanging issue when Bazel failed to upload action inputs
     (bazelbuild#16819)
   + 2f0f3e1:
     [5.4.0] Add integration tests for
     --experimental_credential_helper. (bazelbuild#16880)
   + 6d2d68d:
     [5.4.0] Keep credentials cached across build commands. (bazelbuild#16884)
   + 676a0c8:
     Update Bazel to depend on bazelbuild/platforms 0.0.5.
   + 0ea070b:
     Backport recent package metadata and license check capabilities
     from Bazel 6.x. (bazelbuild#16892)
   + b51396a:
     Add 'toolchain' parameter to actions.{run,run_shell} (bazelbuild#16964)
   + 312fcab:
     Release 5.4.0 (2022-12-15)
   + 0c4e292:
     Pin Bazel version to 5.4.0 (bazelbuild#17986)
   + 43dadb2:
     Bump minimum supported macOS versions to 10.13
   + 1f2b3ed:
     Patch zlib to fix compatibility with latest Xcode
   + a35f592:
     Use ctime in file digest cache key (bazelbuild#18115)
   + e6af231:
     [5.4.1] Disable failing tests (bazelbuild#18123)

Incompatible changes:

  - GrpcRemoteDownloader only includes relevant headers instead of
    sending all credentials.

    Closes bazelbuild#16439.
  - analysis_test moved into testing.analysis_test

Important changes:

  - alias() can now select() directly on constraint_value()

    Fixes bazelbuild#13047.

    Closes bazelbuild#14310.
  - Fixed an issue where Bazel could erroneously report a test passes
    in coverage mode without actually running the test.
  - Make protocOpts() publicly accessible.
  - Add coverage configuration fragment, used to expose
    output_generator label.
  - Bazel now no longer includes system headers on macOS in coverage
    reports (bazelbuild#14969).

    Closes bazelbuild#14971.
  - Starlark test rules can use the new inherited_environment
    parameter of testing.TestEnvironment to specify environment
    variables
    whose values should be inherited from the shell environment.

    Closes bazelbuild#14849.
  - none
    RELNOTES:none
  - Enable merging permissions during Android manifest merging with
    the --merge_android_manifest_permissions flag.
  - Added new register_{execution_platforms,toolchains} directives to
    the MODULE.bazel file, to replace the
    {execution_platforms,toolchains}_to_register attributes on the
    module() directive.
  - Add support for fetching RPC credentials from credential helper.

    Progress on bazelbuild#15856

    Closes bazelbuild#15947.
  - `cquery`'s new output mode
    [`--output=files`](https://bazel.build/docs/cquery#files-output)
    lists the output files of the targets matching the query. It
    takes the current value of `--output_groups` into account.

    Closes bazelbuild#15552.
  - Fix for desugaring failure on Bazel+Android+Windows build
    scenario.
  - The new path variable `$(rlocationpath ...)` and its plural form
    `$(rlocationpaths ...)` can be used to expand labels to the paths
    accepted by the `Rlocation` function of runfiles libraries. This
    is the preferred way to access data dependencies at runtime and
    works on all platforms, even when runfiles are not enabled (e.g.,
    on Windows by default).

    Work towards bazelbuild#16124
    Fixes bazelbuild#10923

    Closes bazelbuild#16667.

This release contains contributions from many people at Google, as well as amberdixon, Andreas Fuchs, Benjamin Peterson, Brentley Jones, Chenchu Kolli, Dan Fleming, Danny Wolf, Emil Kattainen, Fabian Meumertzheim, Gowroji Sunil, hvadehra, Juh-Roch, Keith Smiley, Krzysztof Naglik, kshyanashree, Niyas Sait, Noa Resare, Oliver Eikemeier, oquenchil, Peter Mounce, Philipp Schrader, Ryan Beasley, Thi Doãn, Xùdōng Yáng, Yannic, Zhongpeng Lin.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@meteorcloudy meteorcloudy meteorcloudy approved these changes

@ShreeM01 ShreeM01 Awaiting requested review from ShreeM01

Assignees
No one assigned
Labels
team-Core Skyframe, bazel query, BEP, options parsing, bazelrc team-ExternalDeps External dependency handling, remote repositiories, WORKSPACE file.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@fmeum @keertk @meteorcloudy @lijunsong