Incompatibility between musl and crates linking to shared libraries

[ath-inactive-account]

I'm trying to rewrite kmscube in rust, targetting an environment with musl as libc implementation.
kmscube's goal is to demonstrate the use of opengl without any compositor. It relies on mesa3d's libgbm, which is present as a shared library in my musl environment, at /usr/lib/libgbm.so. There is a rust crate wrapping libgbm : gbm.rs. I'm using the crate as follows:

extern crate gbm;

use gbm::Device as GBMDevice;
use std::fs::OpenOptions;

fn main() {
	let card = {
		let mut options = OpenOptions::new();
		options.read(true);
		options.write(true);
		options.open("/dev/dri/card0").unwrap()
	};

	println!("pre-gbm");
	let _card = GBMDevice::new(card).unwrap();
	println!("post-gbm");
}

Using the gnu (non-musl) target, this happens:

cargo run
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/azimuth`
pre-gbm
post-gbm

Using the musl target, this happens:

cargo run --target=x86_64-unknown-linux-musl
    Finished dev [unoptimized + debuginfo] target(s) in 0.01s
     Running `target/debug/azimuth`
pre-gbm
Segmentation fault (core dumped)

And this SIGSEGV is my problem.

Analysis

readelf reveals the dynamic section of each file:

GNU

 0x0000000000000001 (NEEDED)             Shared library: [libEGL.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libdl.so.2]
 0x0000000000000001 (NEEDED)             Shared library: [libpthread.so.0]
 0x0000000000000001 (NEEDED)             Shared library: [libgcc_s.so.1]
 0x0000000000000001 (NEEDED)             Shared library: [libc.so.6]
 0x0000000000000001 (NEEDED)             Shared library: [ld-linux-x86-64.so.2]
[numeric entries]
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
[numeric entries]

MUSL

 0x0000000000000001 (NEEDED)             Shared library: [libgbm.so.1]
[numeric entries]
 0x000000006ffffffb (FLAGS_1)            Flags: NOW PIE
[numeric entries]

As well as the program headers:

GNU

  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  PHDR           0x0000000000000040 0x0000000000000040 0x0000000000000040
                 0x0000000000000310 0x0000000000000310  R      0x8
  INTERP         0x0000000000000350 0x0000000000000350 0x0000000000000350
                 0x000000000000001c 0x000000000000001c  R      0x1
      [Requesting program interpreter: /lib64/ld-linux-x86-64.so.2]
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x00000000000040e0 0x00000000000040e0  R      0x1000
  LOAD           0x0000000000005000 0x0000000000005000 0x0000000000005000
                 0x00000000000265a5 0x00000000000265a5  R E    0x1000
  LOAD           0x000000000002c000 0x000000000002c000 0x000000000002c000
                 0x0000000000009d58 0x0000000000009d58  R      0x1000
  LOAD           0x0000000000036320 0x0000000000037320 0x0000000000037320
                 0x0000000000001d18 0x0000000000001e98  RW     0x1000
  DYNAMIC        0x00000000000377a8 0x00000000000387a8 0x00000000000387a8
                 0x0000000000000240 0x0000000000000240  RW     0x8
  NOTE           0x0000000000000370 0x0000000000000370 0x0000000000000370
                 0x0000000000000020 0x0000000000000020  R      0x8
  NOTE           0x0000000000000390 0x0000000000000390 0x0000000000000390
                 0x0000000000000044 0x0000000000000044  R      0x4
  TLS            0x0000000000036320 0x0000000000037320 0x0000000000037320
                 0x0000000000000000 0x0000000000000098  R      0x20
  GNU_PROPERTY   0x0000000000000370 0x0000000000000370 0x0000000000000370
                 0x0000000000000020 0x0000000000000020  R      0x8
  GNU_EH_FRAME   0x0000000000030158 0x0000000000030158 0x0000000000030158
                 0x0000000000000b9c 0x0000000000000b9c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x0000000000036320 0x0000000000037320 0x0000000000037320
                 0x0000000000001ce0 0x0000000000001ce0  R      0x1

MUSL

  Type           Offset             VirtAddr           PhysAddr
                 FileSiz            MemSiz              Flags  Align
  LOAD           0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000006e10 0x0000000000006e10  R      0x1000
  LOAD           0x0000000000007000 0x0000000000007000 0x0000000000007000
                 0x00000000000420eb 0x00000000000420eb  R E    0x1000
  LOAD           0x000000000004a000 0x000000000004a000 0x000000000004a000
                 0x000000000000fda4 0x000000000000fda4  R      0x1000
  LOAD           0x000000000005a060 0x000000000005b060 0x000000000005b060
                 0x0000000000002100 0x0000000000003d08  RW     0x1000
  DYNAMIC        0x000000000005b788 0x000000000005c788 0x000000000005c788
                 0x0000000000000140 0x0000000000000140  RW     0x8
  NOTE           0x0000000000000270 0x0000000000000270 0x0000000000000270
                 0x0000000000000024 0x0000000000000024  R      0x4
  TLS            0x000000000005a060 0x000000000005b060 0x000000000005b060
                 0x0000000000000000 0x00000000000000d8  R      0x20
  GNU_EH_FRAME   0x0000000000051e24 0x0000000000051e24 0x0000000000051e24
                 0x000000000000125c 0x000000000000125c  R      0x4
  GNU_STACK      0x0000000000000000 0x0000000000000000 0x0000000000000000
                 0x0000000000000000 0x0000000000000000  RW     0x10
  GNU_RELRO      0x000000000005a060 0x000000000005b060 0x000000000005b060
                 0x0000000000001fa0 0x0000000000001fa0  R      0x1

Using strace, I was able to see that the musl binary never tries to reach for libgbm.
The function call at line 15 results in the use of the symbol gbm_create_device, which was never resolved and points to NULL.
→ segmentation fault.

Interpretation

Upon inspection of the musl target specification in rust-lang/rust, I understand that the musl target systematically produces static binaries. Further in the targets code, the static_position_independent_executables target option is converted to a command line argument for gcc: -static.

From the GCC manual:

-static: On systems that support dynamic linking, this overrides -pie and prevents linking with the shared libraries.

I think GCC achieves this by simply removing the code relocating imported symbols, leaving them unresolved. If we try to use them, we get a segfault.

Does this mean that the musl target is incompatible with any crate that links with shared library at runtime ?
My motivation for using musl is not getting fully static binaries, I just like that it's lightweight and simple. If this incompatibility exists, I'd be happier if I could disable the static_position_independent_executables as a feature, so I could use all crates with musl.

I'm reporting this as a bug because rustc does not warn the developer that using crates linking to a shlib will lead to segfaults.

Meta

rustc --version --verbose:

rustc 1.50.0 (cb75ad5db 2021-02-10)
binary: rustc
commit-hash: cb75ad5db02783e8b0222fee363c5f63f7e2cf5b
commit-date: 2021-02-10
host: x86_64-unknown-linux-gnu
release: 1.50.0

[ath-inactive-account]

I will try to build the musl target with static_position_independent_executables set to false to see if the dynamic linking takes place.

[ath-inactive-account]

This stackoverflow question has good information on the subject of static pie and relocation.

[ath-inactive-account]

Although this is not solved, I solved it for my case. In config.toml, I put this rustc flag:

[build]
rustflags = [ "-C", "target-feature=-crt-static" ]

It looks like this overrides the static_position_independent_executables target configuration.
And while it didn't work on my mixed-libc environment, it worked on my musl-only environment.
Now the binary links to musl. I'm not sure exactly which librariy is dynamically linked and which is statically linked, but at least libgbm symbols are now resolved at runtime.

I don't think I should close the issue, as I don't fully understand how the problem was solved. I welcome any explanation.

[mati865]

Were you building the executable on musl based system (like Alpine or musl variant of Void Linux)?
From the description it seems you are trying to make musl binary linked to shared library that is linked to glibc.

[ath-inactive-account]

Were you building the executable on musl based system (like Alpine or musl variant of Void Linux)?
From the description it seems you are trying to make musl binary linked to shared library that is linked to glibc.

I was trying to build my crate on two different systems. Initially both would segfault.
First one has arch-linux with glibc but also musl in /usr/lib/musl/lib.
Second one is a buildroot embedded linux distribution with musl in /usr/lib.
The config.toml trick worked on the buildroot system immediately (to my surprise), but it was generating a lot of problems on the arch system which I was first testing on.

The mess between musl-libc and glibc on the arch system must have been the reason for all the subsequent problems.

[mati865]

The config.toml trick worked on the buildroot system immediately (to my surprise)

When cross compiling musl target it defaults to static linking yet gbm.rs somehow got linked dynamically (not sure if gbm.rs bug or what). This didn't work because statically linked musl will not load any dynamic libraries.
target-feature=-crt-static made musl to be linked dynamically (it's default when not cross compiling, i.e. when using musl host) so it could load dynamic libraries.

but it was generating a lot of problems on the arch system which I was first testing on.

You were trying to mix glibc and musl libraries which is not supposed to work.

[ath-inactive-account]

Thanks for the explanation.

When cross compiling musl target it defaults to static linking yet gbm.rs somehow got linked dynamically (not sure if gbm.rs bug or what).

I guess that's the real bug then. The crate which links to libgbm is, to be precise, the gbm-sys subcrate. It uses bindgen and its code is there. The link attribute is located here.

[mati865]

Thanks for the explanation.

When cross compiling musl target it defaults to static linking yet gbm.rs somehow got linked dynamically (not sure if gbm.rs bug or what).

I guess that's the real bug then. The crate which links to libgbm is, to be precise, the gbm-sys subcrate. It uses bindgen and its code is there. The link attribute is located here.

So it's gbm-sys bug since it tells Rust to always link shared libgbm: https://doc.rust-lang.org/reference/items/external-blocks.html#the-link-attribute

[ath-inactive-account]

So it's gbm-sys bug since it tells Rust to always link shared libgbm: https://doc.rust-lang.org/reference/items/external-blocks.html#the-link-attribute

I don't understand, the link attribute should select the appropriate link type if the kind key is not specified. An error should be reported if the linking cannot happen, way before it segfault, don't you think ? In which case it's a bug from rustc.

[mati865]

I don't understand, the link attribute should select the appropriate link type if the kind key is not specified.

AFAIK it cannot do it because of backwards compatibility. If you look at this text:

The kind key is an optional value which specifies the kind of library with the following possible values:

• dylib — Indicates a dynamic library. This is the default if kind is not specified.

Empty kind means dynamic linking.

[ath-inactive-account]

Oh, I see.

An error should be reported if the linking cannot happen, way before it segfault, don't you think ? In which case it's a bug from rustc.

Should I leave the issue open for this part ?

[mati865]

An error should be reported if the linking cannot happen, way before it segfault, don't you think ? In which case it's a bug from rustc.

Should I leave the issue open for this part ?

I'll leave it to somebody more experienced like @petrochenkov but IMO this would be hard to take any action here because some libc implementations allow to mix static libc with shared libraries.

[12101111]

I think this is another duplicate of #71647, #81923, #82912 and #39998

[Yuri6037]

The problem also touches cc crate based sys libraries which requires a c++ runtime to work.

Without the workarround provided in the begining of this issue c++/c/rust binaries are generated with a broken path to the dynamic linker (ld-linux instead of ld-musl).

[ghost]

Although this is not solved, I solved it for my case. In config.toml, I put this rustc flag:

[build]
rustflags = [ "-C", "target-feature=-crt-static" ]

It looks like this overrides the static_position_independent_executables target configuration. And while it didn't work on my mixed-libc environment, it worked on my musl-only environment. Now the binary links to musl. I'm not sure exactly which librariy is dynamically linked and which is statically linked, but at least libgbm symbols are now resolved at runtime.

I don't think I should close the issue, as I don't fully understand how the problem was solved. I welcome any explanation.

This solved the problem for me too!

[jbg]

@mati865

target-feature=-crt-static made musl to be linked dynamically (it's default when not cross compiling, i.e. when using musl host)

I don't think this is the default when not cross-compiling, even though it should be. The -musl targets as shipped by Rust always default to linking musl libc statically, even when host and target are both the same -musl triple. Alpine Linux patches their Rust package to fix this.

[mati865]

@jbg I haven't been using musl at all since that comment so I have no idea on how things look right now.

I don't think this is the default when not cross-compiling, even though it should be.

I don't even know if this has never been the case or something has changed.