macOS-only runtime dyld error with dynamically-linked binary indirectly depending on linkstatic=True library

[bjacob]

Summary: See attached minimal testcase. On macOS, Bazel produces a binary that fails at runtime to dynamically link to its dependency. This happens in the following scenario: a binary "c" with linkstatic=False depends on a library "b" with linkstatic=False, itself depending on a library "a" with linkstatic=True. Each of these aspects is necessary to reproduce the issue:

• Seems macOS-specific: the testcase runs fine on Linux.
• If "c" depends directly on "a" instead of through the intermediate "b", the error disappears.
• If "c" is linkstatic=True, the error disappears.
• If "a" is linkstatic=False, the error disappears.

Minimal testcase attached: bazel_link_error.tar.gz

BUILD file from the testcase pasted here for your convenience:

# "a" is a library with linkstatic=True
cc_library(
  name = "a",
  srcs = ["a.cc"],
  hdrs = ["a.h"],
  linkstatic = True,  # Omitting this avoids the link error in "c"
)

# "b" has linkstatic=False (default) and wraps the library "a"
cc_library(
  name = "b",
  srcs = ["b.cc"],
  hdrs = ["b.h"],
  deps = [":a"],
  # Setting linkstatic=True avoids the link error in "c"
)

# cc_test is similar to cc_binary except that linkstatic defaults to
# False in cc_test, vs. True in cc_binary. This testcases reproduces
# also with cc_binary with linkstatic=False.
cc_test(
  name = "c",
  srcs = ["c.cc"],
  deps = [":b"],
)

To reproduce, untar the archive and run //:c

tar xfzv bazel_link_error.tar.gz
cd bazel_link_error
bazel run //:c

Actual result:

Executing tests from //:c
-----------------------------------------------------------------------------
dyld: lazy symbol binding failed: Symbol not found: __Z1av
  Referenced from: /private/var/tmp/_bazel_benoitjacob/b022d434d29bde6eb5304b94cf8f92cb/execroot/__main__/bazel-out/darwin-fastbuild/bin/c.runfiles/__main__/_solib_darwin_x86_64//liblibb.so
  Expected in: flat namespace

dyld: Symbol not found: __Z1av
  Referenced from: /private/var/tmp/_bazel_benoitjacob/b022d434d29bde6eb5304b94cf8f92cb/execroot/__main__/bazel-out/darwin-fastbuild/bin/c.runfiles/__main__/_solib_darwin_x86_64//liblibb.so
  Expected in: flat namespace

[oquenchil]

@keith @googlewalt Hello Keith and Walt,

I'm not sure if what's happening here is a Bazel bug or a limitation of linkers on macOS. This is the command line I got on Linux:

-o
bazel-out/k8-fastbuild/bin/c
-Wl,-rpath,$ORIGIN/_solib_k8/
-Lbazel-out/k8-fastbuild/bin/_solib_k8
-Wl,-S
-fuse-ld=/usr/AAAAAA/clang11/bin/ld.lld
-Wl,-no-as-needed
-Wl,-z,relro,-z,now
-B/usr/AAAAAAAA/clang11/bin
bazel-out/k8-fastbuild/bin/_objs/c/c.pic.o
-llibb
-Wl,--start-lib
bazel-out/k8-fastbuild/bin/_objs/a/a.pic.o
-Wl,--end-lib
-lstdc++
-lm

This is the command line my colleague got on macOS:

-o
bazel-out/darwin-fastbuild/bin/c
-Wl,-rpath,@loader_path/_solib_darwin/
-Lbazel-out/darwin-fastbuild/bin/_solib_darwin
bazel-out/darwin-fastbuild/bin/_objs/c/c.pic.o
-llibb
bazel-out/darwin-fastbuild/bin/liba.a
-Wl,-S
-undefined
dynamic_lookup
-headerpad_max_install_names

I see that liba.a is passed rather than the object file but that should not matter (that's enabled with the feature SUPPORTS_START_END_LIB). What is happening here is that a shared object (libb) depends on a which is not linked dynamically against it but is linked into the binary (c). Since when we build b we don't have the binary yet, we don't even know what it would be called, then we can't make a be a dynamic dependency of b. On Linux this works, you don't have to link a dynamically to b, it will find the symbols later in the binary but I know that this doesn't work on Windows. Is it the same case on macOS?

If that limitation exists, then this just won't work. If the reason why you have this setup is because you have a precompiled archive that you don't control and they won't provide you with a shared object instead, then what you can do is wrap the precompiled archive in a cc_binary(linkshared=1). If that's not the case and you are compiling everything from source, why do you need this setup?

# "a" is a library with linkstatic=True
cc_binary(
  name = "a",
  srcs = ["liba.a"], # Assuming you just have liba.a
  linkshared = True,
)
cc_library(
  name = "dynamic_a",
  hdrs = ["a.h"],
  srcs = ["a"],
)

# "b" has linkstatic=False (default) and wraps the library "a"
cc_library(
  name = "b",
  srcs = ["b.cc"],
  hdrs = ["b.h"],
  deps = [":dynamic_a"],
  # Setting linkstatic=True avoids the link error in "c"
)

# cc_test is similar to cc_binary except that linkstatic defaults to
# False in cc_test, vs. True in cc_binary. This testcases reproduces
# also with cc_binary with linkstatic=False.
cc_test(
  name = "c",
  srcs = ["c.cc"],
  deps = [":b"],
)

[keith]

I see that liba.a is passed rather than the object file but that should not matter

this isn't particularly true, I don't fully understand your repro case, but can you try to use alwayslink / force_load with your library? more info on another case: bazelbuild/rules_apple#332 (comment)

[googlewalt]

This essentially boils down to:

clang++ c.o libb.so liba.a

For this test case to work, the linker needs to pull in symbols from a into the c binary to satisfy undefined symbols in b. This has to be done by the linker (as opposed to anything bazel can do), and ld64 (Apple's linker) does not appear to support this use case.

If "c" depends directly on "a" instead of through the intermediate "b", the error disappears.

I think the error still remains, for the same reason.

[comius]

It's 2024. This looks like P1 issue since 2021? Is it still relevant?

[googlewalt]

Yeah I think we can close this. I don't think this was a supported use case, and it's WAI.