Bazel CI Flaky src.test.shell.bazel.bazel_determinism_test test_determinism

[sgowroji]

https://buildkite.com/organizations/bazel/analytics/suites/bazel-bazel/tests/01884407-94a5-753d-a416-a54987896134?branch=master

Test Details :

BAZELCI_TASK=macos_arm64 TEST_TARGET=//src/test/shell/bazel:bazel_determinism_test TEST_NAME=test_determinism

 bazel test //src/test/shell/bazel:bazel_determinism_test --test_filter=test_determinism 

Logs:

test_determinism FAILED: Non-deterministic outputs found!.
/private/var/tmp/_bazel_buildkite/723aa3a3beab6a8a2b40efcba5b728ce/sandbox/darwin-sandbox/8498/execroot/_main/bazel-out/darwin_arm64-fastbuild/bin/src/test/shell/bazel/bazel_determinism_test.runfiles/_main/src/test/shell/bazel/bazel_determinism_test:109: in call to test_determinism

CC Greenteam @SalmaSamy

[meteorcloudy]

This looks like the same issue in #4770

[meteorcloudy]

It looks like building xcode-locator is non-determinisitc:

ci@bk-macstudio-10 bazel % cd /private/var/tmp/_bazel_ci/ba891f1787a15980e5a85b9711097c39/execroot/_main
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
962cb8d0d3ad9b7acfe05a43322bb3cafc75b064e89592b5addcbb0bbffe535c  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
1a156c388390c13321ae7b8da7a9aabb78506b31cc784c2354fdeb804a51688b  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
962cb8d0d3ad9b7acfe05a43322bb3cafc75b064e89592b5addcbb0bbffe535c  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
921453e58867c03eceef9972c35ee975fd213c7e2dd97013506c2e93c0b8c57f  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
921453e58867c03eceef9972c35ee975fd213c7e2dd97013506c2e93c0b8c57f  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
1a156c388390c13321ae7b8da7a9aabb78506b31cc784c2354fdeb804a51688b  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
1a156c388390c13321ae7b8da7a9aabb78506b31cc784c2354fdeb804a51688b  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
ci@bk-macstudio-10 _main % /usr/bin/xcrun --sdk macosx clang -mmacosx-version-min=10.13 -fobjc-arc -framework CoreServices       -framework Foundation -arch arm64 -arch x86_64 -Wl,-no_adhoc_codesign -Wl,-no_uuid -o bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator tools/osx/xcode_locator.m &&   env -i codesign --identifier bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator --force --sign - bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator && sha256sum bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator
921453e58867c03eceef9972c35ee975fd213c7e2dd97013506c2e93c0b8c57f  bazel-out/darwin_arm64-fastbuild/bin/tools/osx/xcode-locator

[meteorcloudy]

/cc @keith I found your post: https://www.smileykeith.com/2021/10/05/codesign-m1/, any idea why this is happening?

[meteorcloudy]

We are using Xcode 14.2 in those builds. This doesn't seem to happen after I switch to 15.1

[meteorcloudy]

Hmm, even with Xcode 15.1, it still sometimes fails with

-- Test log: -----------------------------------------------------------
--- /private/var/tmp/_bazel_buildkite/d08d56de6f297708263f80d3af3f674f/sandbox/darwin-sandbox/8928/execroot/_main/_tmp/6f8501687b547ae4024a33a8e713e307/sum1	2024-01-16 10:05:18
+++ /private/var/tmp/_bazel_buildkite/d08d56de6f297708263f80d3af3f674f/sandbox/darwin-sandbox/8928/execroot/_main/_tmp/6f8501687b547ae4024a33a8e713e307/sum2	2024-01-16 10:07:30
@@ -3617 +3617 @@
-bazel-bin/src/bazel c752b343e6fda648a854438e3075bb94fb1b6cc8
+bazel-bin/src/bazel 97226420dcec325a07ee54d6aed9030233ed8b23
@@ -3620 +3620 @@
-bazel-bin/src/install_base_key_jdk_minimal 36f527f2cfb636029d33f8ba78b53ab81cf02bfc
+bazel-bin/src/install_base_key_jdk_minimal 6dd8037b762ca4952c85879d94adbaffc36e9455
@@ -3673 +3673 @@
-bazel-bin/src/main/cpp/client c2845e9a69c3a9330bd0c1531dea00fe76eb512a
+bazel-bin/src/main/cpp/client 44add1ad4992f14230efc95a59411cfca3b59bd4
@@ -19765 +19765 @@
-bazel-bin/src/package_jdk_minimal.zip f51afb3d5e62c26445b7c58ee6a48737807e82da
+bazel-bin/src/package_jdk_minimal.zip 3391d6ad1565ab0526be0dd3b234687d3ee70b09
------------------------------------------------------------------------

See https://buildkite.com/bazel/bazel-bazel/builds/26284#018d118e-e615-4084-a001-e06ee66bb36d

Looks like the clang compiler might be non-determinsitic on macOS?

[keith]

I can't get this to fail on my machine after ~1000 runs with Xcode 15.1. It's possible the host macOS version is part of the problem. If you can provide the 2 differing binaries I can try to debug what differs at least

[keith]

I fixed some version of this in clang already, and I believe apple fixed it in the linker itself too, so I think if it's broken in a similar way it's a new bug

[meteorcloudy]

I does happen a lot less frequent than before, I'll collect the binaries if I manage to reproduce. Thanks!