Add Homebrew libraries to the library search on Apple Silicon/darwin-arm64, or how to find them from other software?

[eregon]

Provide a detailed description of the proposed feature

I wonder why Homebrew does not seem to add the libraries it installs to the system library search path on Apple Silicon/darwin-arm64.
Specifically, the case I care about here is Ruby FFI and other dlopen() users, and how they are supposed to find libraries installed via Homebrew: ffi/ffi#880 (comment)
I'm also interested in the general question: "How should software compiled from source (e.g., ruby) find Homebrew-installed headers+libraries?"

There is some discussion in #9177 and notably #9177 (comment)
But I am not clear on what's really a solution there.

• brew --prefix wget is unusable for FFI purposes as we only get the library name, no the package name.
• brew shellenv does not seem applicable as we need to stay in the same process

What is the motivation for the feature?

Make it easier for Ruby FFI and other dlopen() users to use Homebrew-installed libraries.

How will the feature be relevant to at least 90% of Homebrew users?

By making it easier (ideally automatic with no user intervention) for Ruby FFI and other dlopen() users to use Homebrew-installed libraries (and even software compiled from source in general) to find Homebrew packages, there is less need to know about lots of details (e.g., how to pass the openssl dir for ruby, actually not trivial it's --with-openssl-dir + PKG_CONFIG_PATH) and workarounds to make it work.
My observation is many macOS users in particular seem to not know much about their system, so the less they need to tweak the better chance they do not mess it up (which seems to happen a lot given a lot more report from macOS rather than Linux on most softwares).

What alternatives to the feature have been considered?

Searching a hardcoded directory like /opt/homebrew/lib/ but it feels like a ugly workaround and might not work long-term, plus it does not scale to maintain such exceptions for every platform.
Another way to think about this is every single platform supported by Ruby FFI needs no special search path, except darwin-arm64, how can we avoid it?

[eregon]

There was some discussion in https://github.com/orgs/Homebrew/discussions/1600 related to this, cc @larskanis, but it does not seem to lead to a clear or official solution to me.

[MikeMcQuaid]

Searching a hardcoded directory like /opt/homebrew/lib/ but it feels like a ugly workaround and might not work long-term, plus it does not scale to maintain such exceptions for every platform.

Shell out to brew --prefix and search $(brew --prefix)/lib. Alternatively, hardcode Homebrew's default directories. I don't really see other options here, unfortunately.

[eregon]

Thanks for the quick reply.
To understand better, would it make sense to add /opt/homebrew/lib to e.g. DYLD_FALLBACK_LIBRARY_PATH?
Or Homebrew on darwin-arm64 wants on purpose to be found by no build system/dlopen() unless added explicitly? (to avoid potential conflicts?)

[MikeMcQuaid]

To understand better, would it make sense to add /opt/homebrew/lib to e.g. DYLD_FALLBACK_LIBRARY_PATH?

By Homebrew or by FFI? I'd say "probably" for us and "maybe" for you 😁

We'd need to set it in brew shellenv and I'm not convinced that:

• this is used widely enough to be useful
• this would have no other negative effects

Or Homebrew on darwin-arm64 wants on purpose to be found by no build system/dlopen() unless added explicitly? (to avoid potential conflicts?)

No, this is not intentional.

[eregon]

I meant by Homebrew, but maybe it's interesting to see if that'd work for FFI. I'm not sure that modifications to this env var are considered after the process started, would need to check.

No, this is not intentional.

Interesting, then I think it'd be worth considering if setting DYLD_FALLBACK_LIBRARY_PATH and/or other env vars to make packages installed by Homebrew available by default would make sense (e.g., PATH is already set).
Making Homebrew packages available by default already worked on darwin-amd64 without needing env vars, due to being in a standard /usr/local prefix which is already considered by compilers/linkers/etc, so in some sense there is some regression here on darwin-arm64 in that it's much harder to use Homebrew packages for compiled-from-source software and dlopen().

[MikeMcQuaid]

Interesting, then I think it'd be worth considering if setting DYLD_FALLBACK_LIBRARY_PATH and/or other env vars to make packages installed by Homebrew available by default would make sense (e.g., PATH is already set).

Yup. @Homebrew/maintainers any thoughts here?

I don't think we can rely on brew shellenv alone, here. We'd probably also need a brew doctor check.

Making Homebrew packages available by default already worked on darwin-amd64 without needing env vars, due to being in a standard /usr/local prefix which is already considered by compilers/linkers/etc, so in some sense there is some regression here on darwin-arm64 in that it's much harder to use Homebrew packages for compiled-from-source software and dlopen().

That's fair 👍🏻

[eregon]

this is used widely enough to be useful

Probably a bit of a chicken-and-egg problem, i.e., if brew shellenv would set everything needed to use Homebrew packages then it'd probably see a lot more usage.

this would have no other negative effects

Yes, definitely something to consider, and should system packages be in front or behind Homebrew packages, should a package/library/header exists for both.

[MikeMcQuaid]

Probably a bit of a chicken-and-egg problem, i.e., if brew shellenv would set everything needed to use Homebrew packages then it'd probably see a lot more usage.

Sorry, I don't agree here. I'm thinking partly "people who don't need brew shellenv to set their environment shouldn't be forced to do so" and also "things I run through e.g. VSCode may not run brew shellenv".

[fxcoudert]

Be aware that there are some security restrictions to the use of DYLD_LIBRARY_PATH:

Any dynamic linker (dyld) environment variables, such as DYLD_LIBRARY_PATH, are purged when launching protected processes.

[carlocab]

Not a fan of DYLD_[FALLBACK_]LIBRARY_PATH, and neither is Apple:

❯ FOO=foo /bin/bash -c 'echo ${FOO-unset}'
foo
❯ DYLD_LIBRARY_PATH=foo /bin/bash -c 'echo ${DYLD_LIBRARY_PATH-unset}'
unset
❯ DYLD_FALLBACK_LIBRARY_PATH=foo /bin/bash -c 'echo ${DYLD_FALLBACK_LIBRARY_PATH-unset}'
unset

This is basically the security restriction that @fxcoudert references above.

I am open to adding $HOMEBREW_PREFIX/lib to binary RPATH entries as needed to specific formulae. This will allow those formulae to call dlopen on libraries that are linked into $HOMEBREW_PREFIX/lib. It doesn't help with things users have built themselves, but I suppose we can add documentation suggesting adding the necessary RPATH entries wherever needed.

[eregon]

Is there something like an actual system library search path on macOS, which can be set globally and permanently?
Something like /etc/ld.so.conf on Linux.

[fxcoudert]

@eregon no I am pretty sure there is not

[eregon]

For Ruby FFI, we'll probably search directories for now (since there seems to be no way to make dlopen("libfoo.dylib") find Homebrew libs), any opinion on what should be the order between:

/usr/lib
/usr/local/lib
/opt/local/lib
/opt/homebrew/lib

Specifically, where should /opt/homebrew/lib be in that list? (the other 3 paths are preexisting and we probably should not change their order for compatibility).
(from ffi/ffi#965 (comment))

[carlocab]

Specifically, where should /opt/homebrew/lib be in that list?

After /usr/lib makes sense to mimic behaviour on Intel macOS. Where you put it after that probably depends on what you (or what you think your users) prioritise.

Putting it before /usr/local/lib might make sense to avoid accidentally picking up libraries from a Rosetta installation of Homebrew. On the other hand, this might annoy users who built things from source themselves with something like ./configure && make install. I suspect there are fewer users who build from source than there are who rely on some sort of package manager install things, though.

---

Something that might make sense to add to brew shellenv on non-/usr/local installs of Homebrew is

CPPFLAGS="-I${HOMEBREW_PREFIX}/include${CPPFLAGS+ ${CPPFLAGS}}"
LDFLAGS="-L${HOMEBREW_PREFIX}/lib -Wl,-rpath,${HOMEBREW_PREFIX}/lib${LDFLAGS+ ${LDFLAGS}}"

though. It doesn't solve your problem, but it might help make it smaller.

[rfc6919]

Not a fan of DYLD_[FALLBACK_]LIBRARY_PATH, and neither is Apple:

[...]

This is basically the security restriction that @fxcoudert references above.

In particular, since env is a sip-protected binary, this fails for any script that shebangs with #!/usr/bin/env whatever like python and lua scripts tend to do. (eg: vsergeev/luaradio#13 (comment))

[Bo98]

Somewhat related discussion (since Python reads all of the DYLD_* variables): https://github.com/orgs/Homebrew/discussions/3424

In particular, since env is a sip-protected binary, this fails for any script that shebangs with #!/usr/bin/env whatever like python and lua scripts tend to do.

This is quite a notable shortcoming.

[github-actions]

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs.

[eregon]

Let's close this, thanks for the discussion.
It seems macOS has no way to globally add a directory in the library search path, and so there is way besides explicitly looking in /opt/homebrew/lib (well, except pkg-config maybe) to find Homebrew libs. I blame Apple for this.
That is an unfortunate regression from /usr/local where it just worked, but that's how it is.
DYLD_* seem full of shortcomings so that seems unusable in practice.