Relative rpaths are used for a pkgconfig dependency on Linux

[oscarbenjamin]

This is similar to gh-3882 but I think different because this is for dependencies found by pkgconfig. This comes from scientific-python/spin#176.

I have made a simple demo repo: https://github.com/oscarbenjamin/rpath_meson

I build gmp as an external dependency and install into a local directory and then try to make a Python extension module that links against it:

PREFIX=$(pwd)/.local

curl -O https://gmplib.org/download/gmp/gmp-6.3.0.tar.xz
tar -xf gmp-6.3.0.tar.xz
cd gmp-6.3.0
  ./configure --prefix=$PREFIX
  make -j
  make install
cd ..

PKG_CONFIG_PATH=$PREFIX/lib/pkgconfig meson setup build

meson compile -C build
meson install -C build --destdir ../build-install

export PYTHONPATH=$(pwd)/build-install/usr/local/lib/python3.12/site-packages

python -c 'import meson_test; print(meson_test.pow1000(2))'

This works fine on macos but fails on Linux. More precisely what fails is that at runtime it loads the system libgmp.so (a different version) rather than the one in .local so at least on this system it seems to work but is not correct.

We find gmp with:

gmp = dependency(
  'gmp',
  version: '>= 6.3.0',
)

The dependency is found using pkgconfig. I give an absolute path in PKG_CONFIG_PATH which finds gmp.pc which also has absolute paths:

$ cat .local/lib/pkgconfig/gmp.pc 
prefix=/home/oscar/current/active/rpath_meson/.local
exec_prefix=${prefix}
includedir=${prefix}/include
libdir=${exec_prefix}/lib

Name: GNU MP
Description: GNU Multiple Precision Arithmetic Library
URL: https://gmplib.org
Version: 6.3.0
Cflags: -I${includedir}
Libs: -L${libdir} -lgmp

However the binary built by meson compile uses a relative rpath:

$ readelf -d build/src/meson_test/_meson_test.cpython-312-x86_64-linux-gnu.so

Dynamic section at offset 0x9d78 contains 26 entries:
  Tag        Type                         Name/Value
 0x0000000000000001 (NEEDED)             Shared library: [libgmp.so.10]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x000000000000001d (RUNPATH)            Library runpath: [$ORIGIN/../../../.local/lib]
...

That relative rpath is stripped out by meson install so after install we can't find libgmp.so any more (actually we get the system libgmp which is not what we want):

$ ldd build/src/meson_test/_meson_test.cpython-312-x86_64-linux-gnu.so
	linux-vdso.so.1 (0x00007ffd0f79b000)
	libgmp.so.10 => /home/oscar/current/active/rpath_meson/build/src/meson_test/../../../.local/lib/libgmp.so.10 (0x00007dc9af8f3000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007dc9af600000)
	/lib64/ld-linux-x86-64.so.2 (0x00007dc9af980000)
$ ldd build-install/usr/local/lib/python3.12/site-packages/meson_test/_meson_test.cpython-312-x86_64-linux-gnu.so 
	linux-vdso.so.1 (0x00007ffd4d3ec000)
	libgmp.so.10 => /lib/x86_64-linux-gnu/libgmp.so.10 (0x00007a1b6015b000)
	libc.so.6 => /lib/x86_64-linux-gnu/libc.so.6 (0x00007a1b5fe00000)
	/lib64/ld-linux-x86-64.so.2 (0x00007a1b60202000)

I don't see why a relative path is being used to an external dependency whose location is given as an absolute path. Certainly my intention at the end is that the binaries should reference my local build of libgmp.so by absolute path which is what happens when running the same on macos rather than Linux.

[oscarbenjamin]

On macos what happens is that we also have a relative rpath but if I understand correctly libgmp is referenced by an absolute path in the binary:

$ otool -l build/src/meson_test/_meson_test.cpython-312-darwin.so
...
Load command 10
          cmd LC_LOAD_DYLIB
      cmdsize 88
         name /Users/enojb/work/dev/rpath_meson/.local/lib/libgmp.10.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 16.0.0
compatibility version 16.0.0
Load command 11
          cmd LC_LOAD_DYLIB
      cmdsize 56
         name /usr/lib/libSystem.B.dylib (offset 24)
   time stamp 2 Thu Jan  1 01:00:02 1970
      current version 1345.100.2
compatibility version 1.0.0
Load command 12
          cmd LC_RPATH
      cmdsize 48
         path @loader_path/../../../.local/lib (offset 12)
...

This relative rpath is also stripped out by meson install but that is okay because the binary still has an absolute path reference to libgmp:

$ otool -L build-install/usr/local/lib/python3.12/site-packages/meson_test/_meson_test.cpython-312-darwin.so
build-install/usr/local/lib/python3.12/site-packages/meson_test/_meson_test.cpython-312-darwin.so:
	/Users/enojb/work/dev/rpath_meson/.local/lib/libgmp.10.dylib (compatibility version 16.0.0, current version 16.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)
$ otool -L build/src/meson_test/_meson_test.cpython-312-darwin.so
build/src/meson_test/_meson_test.cpython-312-darwin.so:
	/Users/enojb/work/dev/rpath_meson/.local/lib/libgmp.10.dylib (compatibility version 16.0.0, current version 16.0.0)
	/usr/lib/libSystem.B.dylib (compatibility version 1.0.0, current version 1345.100.2)

[rgommers]

Just tried on macOS to run the test package build (my bad, didn't read the pre-step), got:

meson.build:10:6: ERROR: Dependency lookup for gmp with method 'pkgconfig' failed: Invalid version, need 'gmp' ['>= 6.3.0'] found '6.2.1'.

Second time I ran it, it did work. Seems like the test installs libgmp to the libdir of the active env 🤔:

% pkg-config --libs gmp
-L/Users/rgommers/mambaforge/envs/scipy-dev/lib -lgmp

Had some other project where this came up recently. I'm honestly still not 100% sure what the exact rules for RPATH rewriting when running meson install are. There's a brief release note on it at https://mesonbuild.com/Release-notes-for-0-55-0.html#rpath-removal-now-more-careful. I do think they should be the same on Linux as on macOS though.

There are 4 cases at least to consider, when seeing -L/path/to/mylib -lmylib in the .pc file. The shared library is located:

1. in the libdir of the system prefix (or somewhere else that is on the compiler's library search path)
2. in a subdir of the system libdir, not on the library search path
3. in some other directory outside of system prefix, not within the the project's source tree
4. in some other directory outside of system prefix, within the the project's source tree

Expected behavior:

1. RPATH entry gets stripped
2. RPATH does not get stripped (remembered from another issue that I can't find back right now, where the decision was that this is probably the right thing to do, because this avoids overriding a conscious decision by the distro maintainers to not put /usr/local/lib on the library search path)
3. ??
4. ?. I think the desired behavior is to strip, because a directory in the project should be part of the project, and handled via install_rpath if needed.

(3) is least clear to me. Not stripping would fix @oscarbenjamin's problem I think, but what is expected here? Not stripping would lead to installed packages that only work locally (a win!) but cannot be distributed as packages, because the path is unlikely to be portable (potential risk!). The alternative is probably to not use -L/path/to/mylib -lmylib but instead to use /path/to/mylib/libmylib.so as that should always yield an absolute rather than relative RPATH entry.

@eli-schwartz would you be able to clarify the rules?

[eli-schwartz]

There are three sources of rpaths.

• coming from LDFLAGS or a dependency(), external to meson, as an -rpath linker flag
• coming from install_rpath directives listed in a meson.build file
• automatically added by meson for the purposes of getting uninstalled (in-tree / devenv) to work out of the box

The first type is "build plus install rpaths", the second type is "install rpaths", and the third type is "build rpaths". Meson doesn't currently try to figure out what would be a good runtime RPATH via heuristics.

[rgommers]

I think this is only about dependency(), - when the .pc file contains -L/path/to/mylib -lmylib. install_rpath isn't used, and it's for dependencies not part of the project being built.

[rgommers]

as an -rpath linker flag

Or maybe that's the missing bit here. Meson itself doesn't change anything, but somehow -L{libdir} -lmylib sometimes but not always writes an RPATH into the produced binary. And adding -Wl,-rpath is needed to make this reliable.

[rgommers]

Yep indeed, I think this is why I had a hard time to reproduce the problem: it's conda's environment activation that is adding -Wl,-rpath,/path/to/my/libdir/ to LDFLAGS. That is why I thought that there's a difference regarding where the library is located.

automatically added by meson for the purposes of getting uninstalled (in-tree / devenv) to work out of the box

This is muddying the waters just slightly - it can explain why an editable install with meson-python worked for @oscarbenjamin but it didn't through spin: the former points at the build dir, the latter at the install dir.

I think I get it now. The next step is: how to actually handle the case of a library being installed into some random directory and its .pc file only containing -L/path/to/dir -lmylib. I suspect that that cannot work when installing a package linking against that library on the local machine.

xref scipy/scipy#20585 (review) for pretty much the same issue. There we actually have control over the content of the .pc file, so changing it to Libs: /path/to/dir/mylib.so should work. In the more general case, perhaps export LDFLAGS=-Wl,-rpath=/path/to/dir is the more general solution.

[oscarbenjamin]

Is the solution to use install_rpath but conditionally somehow?

In context I only use this in a development environment. In a normal install the dependencies are either installed system wide or were vendored by auditwheel et al that fix up the rpaths themselves. It seems that meson is getting confused by the fact that spin "installs" into a directory in the development environment. It isn't really a "normal" install so the reasons for stripping rpaths don't apply I think.

[eli-schwartz]

If you've ever read the autotools libtool.m4 source code, it makes for some fascinating reading. The lengths it goes to in order to find the system library load path are... excessive. It even contains a file format parser for ld.so.conf (which doesn't actually document its file format, very fun).

I have pondered teaching this to meson as well. It's a nontrivial endeavor. :(

[oscarbenjamin]

The lengths it goes to in order to find the system library load path are... excessive.

I'm not sure how that relates to the problem at hand. I told meson the path to the pkgconfig files and they told meson some other paths. Then meson has all the needed paths but it strips them out during the install. The question here is not finding the paths but just deciding what to do with them in context.

[eli-schwartz]

And the libtool code I refer to is all about figuring out which paths are duplicates of the system lookup path (which isn't supposed to be included in installed rpaths) and which ones are outside of ld.so.conf and therefore come from third-party dependencies installed to unusual locations.

[rgommers]

That makes sense.

Is the solution to use install_rpath but conditionally somehow?

So this is not a possible solution, for two reasons: the dependency() return value doesn't allow you to access the needed path, and even if it did there's still the same problem that we don't know the loader search path so the "conditional" cannot be done.

Short of what @eli-schwartz has contemplated doing (which I believe is nontrivial indeed), I think there are multiple ways to go about this now:

1. Change the .pc file to: Libs: -L{libdir} -lmylib -Wl,-rpath={libdir}
2. Change the .pc. file to: Libs: /{libdir}/libmylib.so
3. Set LDFLAGS to -Wl,-rpath=/path/to/libdir
4. Set LD_LIBRARY_PATH in the shell when you try to run the built project (meh, not robust to opening a new shell)

I do think they should be the same on Linux as on macOS though.

I haven't actually checked, but I can believe that the behavior is currently different as @oscarbenjamin says. The fix_elf and fix_darwin code paths are completely separate here:

meson/mesonbuild/scripts/depfixer.py

Line 400 in a0ff145

def fix_darwin(fname: str, rpath_dirs_to_remove: T.Set[bytes], new_rpath: str, final_path: str, install_name_mappings: T.Dict[str, str]) -> None:

So possibly a "build rpath" is not being stripped out on macOS.

[eli-schwartz]

We recently fixed Darwin to align closer to what Linux does, in order to fix a bug report where Darwin was deleting many more rpaths than Linux, including some needed ones.

[oscarbenjamin]

Possibly related is this issue about rpath handling on macos: #2121

[oscarbenjamin]

Is the solution to use install_rpath but conditionally somehow?

So this is not a possible solution, for two reasons: the dependency() return value doesn't allow you to access the needed path, and even if it did there's still the same problem that we don't know the loader search path so the "conditional" cannot be done.

I was thinking that the conditional part would be something explicitly opted into by the person running meson like

meson setup build --add-pkgconfig-path-rpaths

(I'm not sure exactly what an option like this should be called or what its behaviour in full generality should be.)

The option to add the rpaths could be a general meson option or a project-specific option. Then spin could tell meson to add or at least not strip out the rpaths when installing into spin's dev install directory.

In the spin case I don't think it makes sense to remove the rpaths apart from when building a wheel. Even then I'm not sure it makes sense because the built wheel is not expected to be relocatable until you run auditwheel which will fix up the rpaths.

What does make sense is that the person running meson install etc should be able to tell meson to add/keep the rpaths for the things that they want the installed project to link to. It should not be necessary for meson to try to guess how to handle these things like I imagine that libtool does.

[oscarbenjamin]

I was thinking that the conditional part would be something explicitly opted into by the person running meson like

I think that people in gh-2121 are asking for the opposite behaviour which is also why it makes sense for this to be an option under control of the person doing the installing.

[rgommers]

Possibly related is this issue about rpath handling on macos: #2121

Very much related indeed, good find. It explained a few things in the discussion that I wish I had known earlier.

2. Change the .pc. file to: Libs: /{libdir}/libmylib.so

Back to the original reproducer in the issue description. I confirmed that my options 1-4 above work. E.g., editing .local/lib/pkgconfig/gmp.pc so the Libs line reads:

Libs: -L${libdir} -lgmp -Wl,-rpath=${libdir}

makes things work with the demo repo on Linux.

The reason that things already work on macOS is that the "install name" of libgmp.dylib as installed to .local is an absolute path. That absolute path gets inherited by the Python extension module built in your demo project (see below for more on that), which is why the absolute path appears in your comment above. If the install name had been @rpath/libgmp.dylib, it would have needed the same -Wl,-rpath=${libdir} addition to gmp.pc as is needed on Linux.

I think that people in gh-2121 are asking for the opposite behaviour

Not quite the opposite. It's another thing that is also needed if (and only if) you're setting an RPATH and the external shared library being linked is built by Meson (GMP is built by Make, so it's a different case). The crucial sentence I think is this one: "MacOS has a peculiar and very unusual way of linking, namely that consumers of a library inherit a provider's install_name". Basically, for -Wl,-rpath,/path/to/local/libdir to work, the shared library's install name must be @rpath/libname.dylib.

So, Linux and macOS are indeed different - and that is because linking works differently on macOS vs. Linux. All Meson is doing is temporarily adding RPATH's inside the build dir and then removing them again on install. That can just have a different effect depending on platform as well as editable install vs. regular install to a destdir.

I was thinking that the conditional part would be something explicitly opted into by the person running meson like

meson setup build --add-pkgconfig-path-rpaths

That does seem like a reasonable option to me to avoid this problem. It's pretty much a band-aid until the structural solution ("teach Meson to find the system library load paths" from @eli-schwartz's comment above) materializes (which could take a long time).

[rgommers]

It even contains a file format parser for ld.so.conf (which doesn't actually document its file format, very fun).

I have pondered teaching this to meson as well

I won't look at the libtool.m4 code since it's GPL, but I did have a look at ld.so.conf. All the ldconfig man page has to say about it is "File containing a list of colon, space, tab, newline, or comma-separated directories in which to search for libraries.", which isn't much to go on. On my system (Arch), all I see is:

$ cat /etc/ld.so.conf
# Dynamic linker/loader configuration.
# See ld.so(8) and ldconfig(8) for details.

include /etc/ld.so.conf.d/*.conf
include /usr/lib/ld.so.conf.d/*.conf


$ ls /etc/ld.so.conf.d/
$ ls /usr/lib/ld.so.conf.d/
fakeroot.conf
$ cat /usr/lib/ld.so.conf.d/fakeroot.conf 
/usr/lib/libfakeroot

Debian is only mildly more interesting, due to Multi-Arch support:

$ cat /etc/ld.so.conf
include /etc/ld.so.conf.d/*.conf


$ ls /etc/ld.so.conf.d/
libc.conf  x86_64-linux-gnu.conf

$ cat /etc/ld.so.conf.d/libc.conf 
# libc default configuration
/usr/local/lib


$ cat /etc/ld.so.conf.d/x86_64-linux-gnu.conf 
# Multiarch support
/usr/local/lib/x86_64-linux-gnu
/lib/x86_64-linux-gnu
/usr/lib/x86_64-linux-gnu

I tried every docker container I had lying around, but didn't find anything that wasn't include statements and newline-separated lists of directories.

The man page sentence hints at the prospect of having to deal with annoying differences between platforms. But besides that being tedious, it's probably doable? Python is a lot better than M4 for writing a file parser, and it seems to be a fairly constrained problem - probably doable in <100 LoC? With a strategy of parsing all known formats to obtain the full search path list, and in case of an unknown format emitting a warning and disabling adding RPATHs (to avoid regressions), maybe this can be introduced in a safe way?

[oscarbenjamin]

For now at least this looks like the best option to make this work:

3. Set LDFLAGS to -Wl,-rpath=/path/to/libdir

I presume this can be done with add_project_link_arguments and it could be done conditionally for a development build via a project option.

I presume also that dependency('gmp').get_pkgconfig_variable('something...') can get the needed path.

[rgommers]

I presume also that dependency('gmp').get_pkgconfig_variable('something...') can get the needed path.

True - if you know that the dependency is detected with pkg-config, this should work.

I presume this can be done with add_project_link_arguments and it could be done conditionally for a development build via a project option.

Agreed, seems reasonable. And then it can be done without having to add anything in Meson itself.

[oscarbenjamin]

I've implemented this for python-flint in flintlib/python-flint#135.

I still think that it would be good for meson to have a standard option for this somehow. If you build against some local build of dependencies then at runtime after install you want to use the same local build as well.

[rgommers]

I still think that it would be good for meson to have a standard option for this somehow.

Only if there is no chance to fix it the right way in a reasonable time frame I think. I have too many projects I'd like to complete already, but I'm actually interested to give it a go (and have some holiday time on my hands).