Bazel CI: Downstream projects broken due to the flip of trim_test_configuration

[coeuvre]

https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/488

Caused by ebac27e.

rules_haskell:

(04:03:37) ERROR: /var/lib/buildkite-agent/builds/bk-docker-0c5s/bazel-downstream-projects/rules_haskell/tests/binary-with-sysdeps/BUILD.bazel:8:13: in _haskell_test rule //tests/binary-with-sysdeps:binary-with-sysdeps:
  | Traceback (most recent call last):
  | File "/var/lib/buildkite-agent/builds/bk-docker-0c5s/bazel-downstream-projects/rules_haskell/haskell/private/haskell_impl.bzl", line 77, column 39, in haskell_test_impl
  | return _haskell_binary_common_impl(ctx, is_test = True)
  | File "/var/lib/buildkite-agent/builds/bk-docker-0c5s/bazel-downstream-projects/rules_haskell/haskell/private/haskell_impl.bzl", line 173, column 25, in _haskell_binary_common_impl
  | cc = cc_interop_info(ctx)
  | File "/var/lib/buildkite-agent/builds/bk-docker-0c5s/bazel-downstream-projects/rules_haskell/haskell/cc.bzl", line 166, column 40, in cc_interop_info
  | cc_libraries = get_cc_libraries(cc_libraries_info, [lib for li in cc_common.merge_cc_infos(cc_infos = ccs).linking_context.linker_inputs.to_list() for lib in li.libraries]),
  | File "/var/lib/buildkite-agent/builds/bk-docker-0c5s/bazel-downstream-projects/rules_haskell/haskell/private/cc_libraries.bzl", line 49, column 43, in get_cc_libraries
  | if not cc_libraries_info.libraries[cc_library_key(lib_to_link)].is_haskell
  | Error: key struct(dynamic_library = <generated file _solib_k8/_U@zlib.hs_S_S_Czlib___Uexternal_Szlib.hs/libz.so>, interface_library = None, pic_static_library = None, static_library = <generated file external/zlib.hs/libz.a>) not found in dictionary

Intellij Plugin Aspect:

(04:02:58) ERROR: file 'aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/_objs/simple/simple.pic.o' is generated by these conflicting actions:
  | Label: //aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest:simple
  | RuleClass: cc_test rule
  | Configuration: 3715a7360b5859b95a658a76cbd9cba49eb2b556c7becd1ce5226f94bc139db3, cc0d70a66dc113c4cbc3e158973e848bf4828c1b7923cb5ec27d1b14bd933030
  | Mnemonic: CppCompile
  | Action key: 113387e566d5d69b91fe46ecd9abc833439318d7fabc425bcb1b47f754d2eadb
  | Progress message: Compiling aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/simple/simple.cc
  | PrimaryInput: File:[/var/lib/buildkite-agent/builds/bk-docker-q1cw/bazel-downstream-projects/intellij[source]]aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/simple/simple.cc
  | PrimaryOutput: File:[[<execution_root>]bazel-out/k8-fastbuild/bin]aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/_objs/simple/simple.pic.o
  | Owner information: ConfiguredTargetKey{label=//aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest:simple, config=BuildConfigurationValue.Key[3715a7360b5859b95a658a76cbd9cba49eb2b556c7becd1ce5226f94bc139db3]}, ConfiguredTargetKey{label=//aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest:simple, config=BuildConfigurationValue.Key[cc0d70a66dc113c4cbc3e158973e848bf4828c1b7923cb5ec27d1b14bd933030]}
  | MandatoryInputs: are equal
  | Outputs: are equal
  | (04:02:58) ERROR: com.google.devtools.build.lib.skyframe.ArtifactConflictFinder$ConflictException: com.google.devtools.build.lib.actions.MutableActionGraph$ActionConflictException: for aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/_objs/simple/simple.pic.o, previous action: action 'Compiling aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/simple/simple.cc', attempted action: action 'Compiling aspect/testing/tests/src/com/google/idea/blaze/aspect/cpp/cctest/simple/simple.cc' 

@sdtwigg, since the breakages are described in the commit message, are they indented behaviors for the flip so we should let downstream to update accordingly?

[fweikert]

These errors still show up in the latest downstream build: https://buildkite.com/bazel/bazel-at-head-plus-downstream/builds/1990#ef50ceea-a88d-4401-b746-bd817e0fafc7

katre@ confirmed that setting --notrim_test_configuration fixed the problem.

[sdtwigg]

The rules_haskell breakage is a little odd. I would expect the --trim_test_configuration change to break the build here with an analysis error if it was going to have any effect rather than subtly change the list of files. I definitely would like for them to look into why this happened and try to fix forward.

The c++ action conflict for the Intellij plugin aspect is definitely not surprising. I am a little concerned the fix-forward is non-trivial since this is potentially a case for non-test depending on test that has not come up before and seems fairly legitimate. I would like to get a little more info on what exactly is happening here (but am preemptively going to look into a somewhat exotic fix to the underlying unsharable action issue).

[fweikert]

The rules_haskell failure can be fixed by setting --notrim_test_configuration:

Without the flag:

fwe~/git/rules_haskell:$ USE_BAZEL_VERSION="last_green" /tmp/bazelisk build --build_tag_filters=-requires_nix,-requires_lz4,-requires_shellcheck,-requires_threaded_rts,-dont_test_with_bindist,-dont_test_on_bazelci --test_env=HOME --test_env=USE_BAZEL_VERSION -- //tests/...
2021/04/12 18:07:44 Using unreleased version at commit 19cb1c13f4029bc8c7b33fd590257a514c38ceef
2021/04/12 18:07:44 Downloading https://storage.googleapis.com/bazel-builds/artifacts/ubuntu1404/19cb1c13f4029bc8c7b33fd590257a514c38ceef/bazel...
[...]
ERROR: /usr/local/google/home/fwe/git/rules_haskell/tests/binary-with-sysdeps/BUILD.bazel:8:13: in _haskell_test rule //tests/binary-with-sysdeps:binary-with-sysdeps: 
Traceback (most recent call last):
	File "/usr/local/google/home/fwe/git/rules_haskell/haskell/private/haskell_impl.bzl", line 77, column 39, in haskell_test_impl
		return _haskell_binary_common_impl(ctx, is_test = True)
	File "/usr/local/google/home/fwe/git/rules_haskell/haskell/private/haskell_impl.bzl", line 173, column 25, in _haskell_binary_common_impl
		cc = cc_interop_info(ctx)
	File "/usr/local/google/home/fwe/git/rules_haskell/haskell/cc.bzl", line 166, column 40, in cc_interop_info
		cc_libraries = get_cc_libraries(cc_libraries_info, [lib for li in cc_common.merge_cc_infos(cc_infos = ccs).linking_context.linker_inputs.to_list() for lib in li.libraries]),
	File "/usr/local/google/home/fwe/git/rules_haskell/haskell/private/cc_libraries.bzl", line 49, column 43, in get_cc_libraries
		if not cc_libraries_info.libraries[cc_library_key(lib_to_link)].is_haskell
Error: key struct(dynamic_library = <generated file _solib_k8/_U@zlib.hs_S_S_Czlib___Uexternal_Szlib.hs/libz.so>, interface_library = None, pic_static_library = None, static_library = <generated file external/zlib.hs/libz.a>) not found in dictionary
ERROR: Analysis of target '//tests/binary-with-sysdeps:binary-with-sysdeps' failed; build aborted: Analysis of target '//tests/binary-with-sysdeps:binary-with-sysdeps' failed
INFO: Elapsed time: 63.615s
INFO: 0 processes.
FAILED: Build did NOT complete successfully (178 packages loaded, 15094 target\
s configured)

With the flag:

fwe~/git/rules_haskell:$ USE_BAZEL_VERSION="last_green" /tmp/bazelisk build --build_tag_filters=-requires_nix,-requires_lz4,-requires_shellcheck,-requires_threaded_rts,-dont_test_with_bindist,-dont_test_on_bazelci --test_env=HOME --test_env=USE_BAZEL_VERSION --notrim_test_configuration  -- //tests/...
[...]
INFO: Elapsed time: 488.951s, Critical Path: 466.29s
INFO: 5023 processes: 2997 internal, 2015 linux-sandbox, 11 local.
INFO: Build completed successfully, 5023 total actions

[fweikert]

Filed tweag/rules_haskell#1536 to track the rules_haskell issue (since there might be something weird going on in their bzl files).

[fweikert]

The downstream pipeline has been red for over two weeks now :(

@sdtwigg: Can you please state whether the rules_haskell breakage is WAI from a Bazel point of view? IF yes I'dd disable it for now. You also mentioned a fix for IntelliJ - what's the status here?

[meteorcloudy]

https://buildkite.com/bazel/bazel-auto-sheriff-face-with-cowboy-hat/builds/488

IntelliJ Plugin Aspect: IntelliJ Plugin Aspect (:ubuntu: 18.04 (OpenJDK 11))
Culprit Found: 627c16e0e4ae70161c5694a369acd21af87e3d50

The Intellij Plugin Aspect project is actually failing due to another commit: 627c16e, so please ignore it for this issue, I'll file another one for it.

[meteorcloudy]

Oh, sorry, the culprit finder actually identified a previous error that is already fixed #13320, the new error message is indeed caused by this one.

[meteorcloudy]

Looks like there is already a pending fix for rules_haskell tweag/rules_haskell#1545, @sdtwigg can you help figure out a fix for Intellij plugin aspect?

[meteorcloudy]

Ping @sdtwigg

[sventiffe]

Ping @sdtwigg. This issue has been marked as bug and P1, is this still accurate? If so, please update the bug. Else, adjust the priority please.

[comius]

rules_haskell works.
intellij plugin is still broken. For 1 month!

The c++ action conflict for the Intellij plugin aspect is definitely not surprising. I am a little concerned the fix-forward is non-trivial since this is potentially a case for non-test depending on test that has not come up before and seems fairly legitimate. I would like to get a little more info on what exactly is happening here (but am preemptively going to look into a somewhat exotic fix to the underlying unsharable action issue).

@sdtwigg Where is that fix forward?

[comius]

This is now fixed.