Bazel 5.3.0 $(location :foo) disagrees with cc_binary's rpath with remote execution, regression from 5.0.0

[AustinSchuh]

Description of the bug:

I've got a sh_test which calls a binary as located by $(location :foo). See https://github.com/frc971/971-Robot-Code/blob/master/aos/network/sctp_lib_test.sh for the test, and https://github.com/frc971/971-Robot-Code/blob/master/aos/network/BUILD#L586 for the build rule. That test is in an external repository. I'm running it by running bazel test @aos//aos/network:sctp_lib_test from the base repository.

When I upgrade from 5.0 to 5.3 and run the test on a remote execution cluster, I get a test failure saying:

++ LD_PRELOAD=external/aos/aos/network/sctp_lib_shim.so
++ external/aos/aos/network/sctp_lib_test_binary --host=localhost
+ output='external/aos/aos/network/sctp_lib_test_binary: error while loading shared libraries: libexternal_Saos_Saos_Snetwork_Slibsctp_Ulib.so: cannot open shared object file: No such file or directory'

The test is getting run from /var/lib/worker/work/7/exec/bazel-out/k8-opt/bin/external/aos/aos/network/sctp_lib_test.runfiles/repo ($(pwd) at the top of the shell script reports this).

Bazel sets rpath in the binary being run to:

  0x000000000000001d (RUNPATH)      Library runpath: [$ORIGIN/../../../repo/_solib_k8/]

Inside the build file, we ask Bazel for the path to the test binary using $(location :sctp_lib_test_binary) Bazel returns external/aos/aos/network/sctp_lib_test_binary.

When run from this path, the shared libraries aren't discoverable. If I instead hard-code the path to the binary to ../aos/aos/network/sctp_lib_test_binary instead, RPATH now agrees with the path it's being run from and the test passes.

I think the fix for this is going to be to make $(location) return ../aos/path instead of external/aos/path

What's the simplest, easiest way to reproduce this bug? Please provide a minimal example if possible.

Either import the repo above as an external repo and try to run the external test, or write a shell script mirroring it.

Which operating system are you running Bazel on?

Debian Bullseye

What is the output of bazel info release?

release 5.3.0-202208151910+2ee906ec8e

If bazel info release returns development version or (@non-git), tell us how you built Bazel.

#!/bin/bash

# This script builds a Debian package from a Bazel source tree with the
# correct version.
# The only argument is the path to a Bazel source tree.

set -e
set -u

BAZEL_SOURCE="$1"

VERSION="5.3.0-$(date +%Y%m%d%H%M)+$(GIT_DIR="${BAZEL_SOURCE}/.git" git rev-parse --short HEAD)"
OUTPUT="bazel_${VERSION}"

(
cd "${BAZEL_SOURCE}"
bazel build -c opt //src:bazel --embed_label="${VERSION}" --stamp=yes
)

cp "${BAZEL_SOURCE}/bazel-bin/src/bazel" "${OUTPUT}"

echo "Output is at ${OUTPUT}"

What's the output of git remote get-url origin; git rev-parse master; git rev-parse HEAD ?

$ git remote get-url origin; git rev-parse master; git rev-parse HEAD
https://github.com/bazelbuild/bazel
master
fatal: ambiguous argument 'master': unknown revision or path not in the working tree.
Use '--' to separate paths from revisions, like this:
'git <command> [<revision>...] -- [<file>...]'
2ee906ec8e530e6dea76c89621b026c9d4eb6701

### Have you found anything relevant by searching the web?

https://github.com/bazelbuild/bazel/issues/16003 fixed most of our failures.  This is the last one (I'm aware of) left.

### Any other information, logs, or outputs that you want to share?

_No response_

[fmeum]

@AustinSchuh Could you try replacing location with rootpath?

[AustinSchuh]

Phew, that was a pain to test. No difference with rootpath

++ LD_PRELOAD=external/aos/aos/network/sctp_lib_shim.so
++ external/aos/aos/network/sctp_lib_test_binary --host=localhost
+ output='external/aos/aos/network/sctp_lib_test_binary: error while loading shared libraries: libexternal_Saos_Saos_Snetwork_Slibsctp_Ulib.so: cannot open shared object file: No such file or directory'

[fmeum]

I see, thanks for checking. I assume that should be fixed by #16215 then, but I'm not completely sure.

[AustinSchuh]

Got it. Let's let the dust settle on 16215 then (looks like it is still in flight) and I can try again.

[fmeum]

@AustinSchuh Could you try this again with 6.0.0rc4?

[github-actions]

Thank you for contributing to the Bazel repository! This issue has been marked as stale since it has not had any activity in the last 1+ years. It will be closed in the next 90 days unless any other activity occurs. If you think this issue is still relevant and should stay open, please post any comment here and the issue will no longer be marked as stale.

[github-actions]

This issue has been automatically closed due to inactivity. If you're still interested in pursuing this, please post @bazelbuild/triage in a comment here and we'll take a look. Thanks!