Cross compilation errors for x86_64-unknown-linux-musl from v1.72

[deepu105]

Code

The cargo build for x86_64-unknown-linux-musl target started failing with the below errors from Rust version 1.72 onwards (see backtrace for full error). The same codebase compiles without issues on 1.71. Issue seems to be around using the xcb library and all required libs are installed on the host

apt-get install -y -qq pkg-config libssl-dev libxcb1-dev libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev xauth musl-tools

cargo build --release --target x86_64-unknown-linux-musl

# also tried
cargo rustc --release --target x86_64-unknown-linux-musl -- -lXau

Project codebase: https://github.com/kdash-rs/kdash. To reproduce, run this docker file on the cloned repo.

rustc 1.75.0-nightly (e0d7ed1f4 2023-10-01)
0.479    Compiling kdash v0.4.3 (/usr/src/kdash-temp)
120.8 error: linking with `cc` failed: exit status: 1
...
120.8   = note: /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_util.o): in function `_xcb_open_tcp':
120.8           (.text+0x3f2): undefined reference to `__snprintf_chk'
120.8           /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_util.o): in function `xcb_connect_to_display_with_auth_info':
120.8           (.text+0x6cc): undefined reference to `__snprintf_chk'
120.8           /usr/bin/ld: (.text+0x72e): undefined reference to `__strcpy_chk'
120.8           /usr/bin/ld: (.text+0x8c6): undefined reference to `__strcpy_chk'
120.8           /usr/bin/ld: (.text+0x9fe): undefined reference to `__strcpy_chk'
120.8           /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_auth.o): in function `get_authptr':
120.8           (.text+0xaf): undefined reference to `__snprintf_chk'
120.8           /usr/bin/ld: (.text+0xfe): undefined reference to `XauGetBestAuthByAddr'
120.8           /usr/bin/ld: (.text+0x14f): undefined reference to `__snprintf_chk'
120.8           /usr/bin/ld: (.text+0x19a): undefined reference to `__snprintf_chk'
120.8           /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_auth.o): in function `_xcb_get_auth_info':
120.8           (.text+0x316): undefined reference to `XauDisposeAuth'
120.8           /usr/bin/ld: (.text+0x552): undefined reference to `XdmcpWrap'
120.8           /usr/bin/ld: (.text+0x562): undefined reference to `XauDisposeAuth'
120.8           collect2: error: ld returned 1 exit status
120.8           
120.8   = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
120.8   = note: use the `-l` flag to specify native libraries to link
120.8   = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
120.8 
120.8 error: could not compile `kdash` (bin "kdash") due to previous error

I have tried to build this natively on alpine Linux and cross-compilation on arch Linux and fails in both.

Version it worked on

It most recently worked on: 1.71

Version with regression

1.72 stable

commit-hash: d5c2e9c342b358556da91d61ed4133f6f50fc0c3                                                                                                                                                                                                                  
commit-date: 2023-09-13                                                                                                                                                                                                                                                
host: x86_64-unknown-linux-gnu                                                                                                                                                                                                                                         
release: 1.72.1                                                                                                                                                                                                                                                        
LLVM version: 16.0.5

1.75 nightly

rustc 1.75.0-nightly (e0d7ed1f4 2023-10-01)                                                                                                                                                                                                                               
0.419 binary: rustc
commit-hash: e0d7ed1f453fb54578cc96dfea859b0e7be15016                                                                                                                                                                                                                  
commit-date: 2023-10-01                                                                                                                                                                                                                                                
host: x86_64-unknown-linux-gnu                                                                                                                                                                                                                                         
release: 1.75.0-nightly                                                                                                                                                                                                                                                
LLVM version: 17.0.2

Backtrace

Backtrace

[builder 13/13] RUN rustc --version --verbose && RUST_BACKTRACE=1 cargo rustc --release --target x86_64-unknown-linux-musl -- -lXau:                                                                                                                                         
0.419 rustc 1.75.0-nightly (e0d7ed1f4 2023-10-01)                                                                                                                                                                                                                               
0.419 binary: rustc                                                                                                                                                                                                                                                             
0.419 commit-hash: e0d7ed1f453fb54578cc96dfea859b0e7be15016                                                                                                                                                                                                                     
0.419 commit-date: 2023-10-01                                                                                                                                                                                                                                                   
0.419 host: x86_64-unknown-linux-gnu
0.419 release: 1.75.0-nightly
0.421 LLVM version: 17.0.2
0.602    Compiling kdash v0.4.3 (/usr/src/kdash-temp)
121.7 error: linking with `cc` failed: exit status: 1
121.7   |
121.7   = note: LC_ALL="C" PATH="/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin:/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-gnu/bin/self-contained:/usr/local/cargo/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" VSLANG="1033" "cc" "-m64" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/rcrt1.o" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crti.o" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtbeginS.o" "/tmp/rustcacwY7H/symbols.o" "/usr/src/kdash-temp/target/x86_64-unknown-linux-musl/release/deps/kdash-bf16b0078dfec89f.kdash.3a14523d29565840-cgu.0.rcgu.o" "-Wl,--as-needed" "-L" "/usr/src/kdash-temp/target/x86_64-unknown-linux-musl/release/deps" "-L" "/usr/src/kdash-temp/target/release/deps" "-L" "/usr/src/kdash-temp/target/x86_64-unknown-linux-musl/release/build/ring-ac80321802c4ed69/out" "-L" "/usr/src/kdash-temp/target/x86_64-unknown-linux-musl/release/build/openssl-sys-629f07187f0967c8/out/openssl-build/install/lib" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-Wl,-Bstatic" "-lXau" "-lxcb" "-lxcb-render" "-lxcb-shape" "-lxcb-xfixes" "/tmp/rustcacwY7H/libring-f3a6144b616625a2.rlib" "-lunwind" "-lc" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/libcompiler_builtins-8f02a7813ca6535d.rlib" "-Wl,-Bdynamic" "-Wl,--eh-frame-hdr" "-Wl,-z,noexecstack" "-nostartfiles" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib" "-L" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained" "-o" "/usr/src/kdash-temp/target/x86_64-unknown-linux-musl/release/deps/kdash-bf16b0078dfec89f" "-Wl,--gc-sections" "-static-pie" "-Wl,-z,relro,-z,now" "-Wl,-O1" "-nodefaultlibs" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtendS.o" "/usr/local/rustup/toolchains/nightly-x86_64-unknown-linux-gnu/lib/rustlib/x86_64-unknown-linux-musl/lib/self-contained/crtn.o"
121.7   = note: /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_util.o): in function `_xcb_open_tcp':
121.7           (.text+0x3f2): undefined reference to `__snprintf_chk'
121.7           /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_util.o): in function `xcb_connect_to_display_with_auth_info':
121.7           (.text+0x6cc): undefined reference to `__snprintf_chk'
121.7           /usr/bin/ld: (.text+0x72e): undefined reference to `__strcpy_chk'
121.7           /usr/bin/ld: (.text+0x8c6): undefined reference to `__strcpy_chk'
121.7           /usr/bin/ld: (.text+0x9fe): undefined reference to `__strcpy_chk'
121.7           /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_auth.o): in function `get_authptr':
121.7           (.text+0xaf): undefined reference to `__snprintf_chk'
121.7           /usr/bin/ld: (.text+0xfe): undefined reference to `XauGetBestAuthByAddr'
121.7           /usr/bin/ld: (.text+0x14f): undefined reference to `__snprintf_chk'
121.7           /usr/bin/ld: (.text+0x19a): undefined reference to `__snprintf_chk'
121.7           /usr/bin/ld: /usr/lib/gcc/x86_64-linux-gnu/12/../../../x86_64-linux-gnu/libxcb.a(xcb_auth.o): in function `_xcb_get_auth_info':
121.7           (.text+0x316): undefined reference to `XauDisposeAuth'
121.7           /usr/bin/ld: (.text+0x552): undefined reference to `XdmcpWrap'
121.7           /usr/bin/ld: (.text+0x562): undefined reference to `XauDisposeAuth'
121.7           collect2: error: ld returned 1 exit status
121.7           
121.7   = note: some `extern` functions couldn't be found; some native libraries may need to be installed or have their path specified
121.7   = note: use the `-l` flag to specify native libraries to link
121.7   = note: use the `cargo:rustc-link-lib` directive to specify the native libraries to link with Cargo (see https://doc.rust-lang.org/cargo/reference/build-scripts.html#cargorustc-link-libkindname)
121.7 
121.7 error: could not compile `kdash` (bin "kdash") due to previous error
------
Dockerfile:31
--------------------
  29 |     RUN apt-get install -y -qq pkg-config libssl-dev libxcb1-dev libxcb-render0-dev libxcb-shape0-dev libxcb-xfixes0-dev xauth
  30 |     
  31 | >>> RUN rustc --version --verbose && RUST_BACKTRACE=1 cargo rustc --release --target x86_64-unknown-linux-musl -- -lXau
  32 |     
  33 |     # -----------------------------
--------------------
ERROR: failed to solve: process "/bin/sh -c rustc --version --verbose && RUST_BACKTRACE=1 cargo rustc --release --target x86_64-unknown-linux-musl -- -lXau" did not complete successfully: exit code: 101
make: *** [Makefile:37: docker] Error 1

[apiraino]

We have a few open issues related to dynamic linking when compiling with musl, I feel also this one could also be related.

@deepu105 can you try what is suggested in this comment? Does it help?

[saethlin]

You are cross-compiling to a musl target, from a gnu distro. The usual reason to use one of the existing -musl targets is to produce a statically linked executable (i.e. so that it is portable/self-contained), but your -musl executables aren't.

How do you expect the executables you are producing to be linked?

[deepu105]

@saethlin I'm quite new to the MUSL world. I saw many popular Rust tools building for MUSL this way so I followed the same and afaik, the built binaries were working fine until this release in the linux distros I tried (Arch, ubuntu, Fedora). I assumed that is the purpose of cross compiling, if not what is the right way to build MUSL binaries?

[bjorn3]

If you are depending on dynamic libraries, they need to be installed on the host system for the correct libc. In your case you depend on the dynamic libraries for X11, so you need the musl versions of those, which I don't think non-musl based distros ship. You need them both at compile time and runtime. Your user almost certainly doesn't have the musl version of these libraries installed.

There are two reasons to use musl:

• You want to produce a completely statically linked library portable to multiple distros.
• You want to run on a musl based linux distro rather than a glibc based distro.

The second reason is almost certainly not the case for you as you want to run on Arch, Ubuntu and Fedora which are all glibc based. As for the first reason, this only works if you are fully statically linked. If you have a GUI application, unless you use a GUI toolkit which doesn't use the GPU (which rules out both GTK and QT), you pretty much can't statically link. You need to dynamically link to the GPU driver, which on a glibc based distro is compiled for glibc and thus can't be loaded by a musl based program.

[Adesoji1]

@deepu105 , While a permanent solution is being sought:

Consider sticking with Rust 1.71 for this specific project if feasible.
If only certain parts of the project need xcb, consider splitting the project into multiple parts and avoiding cross-compilation for the parts that don't need xcb.

[deepu105]

@Adesoji1 I'll stick to 1.71 for now. Thank you

[saethlin]

I don't know how to exercise the xcb functionality in here, but I'd be rather surprised if it works in these musl builds. As far as I can tell you are building Frankenstein binaries: #82912

There is no way to build portable binaries without statically linking the whole thing together. If running ldd reports anything other than

	statically linked

then your binary is not portable across Linuxes.

[deepu105]

@saethlin @bjorn3 I'm definitely a newbie when it comes to static linking or MUSL. I was using the musl builds for brew formul and just did ldd on the binaries and they are indeed not statistically linked as they seem to refer to system libs. Does that mean using cargo to cross compile to MUSL actually doesn't provide a static linked binary? Then what is the purpose of cross compilation?

Also is it correct to assume that even if I build for MUSL from a native MUSL distro like Alpine, the produced binary won't work for XCB?

[bjorn3]

If you build on a musl based distro the executable will work just fine on musl based distros provided that you tell rustc to dynamically link libc. The executable won't work on glibc base distros then though. You can't statically link libxcb so dynamically linking everything is the only option. Dynamically linking to musl libc requires musl libc at runtime.

You can only produce executables portable to glibc based distros using musl if you are statically linking everything, which in your case is not possible due to your dependency on the XCB dynamic library. I strongly recommend just using the glibc target and if you want compatibility with distros using an older glibc versiln, compiling on the distro with the oldest glibc version you want to support. The executable will run fine on systems using a newer glibc version than used when compiling. Just not on older glibc versions.

[Amanieu]

This is most likely due to #111698: previously rustc would link to dynamic libraries while trying to build a static binary, which led to all sorts of problems, including crashes.

Now we properly only link to static libraries when building a static binary, which is the default for the musl target. To build a dynamic binary which can link to dynamic libraries, you need to use RUSTFLAGS="-Ctarget-feature=-crt-static".

[sanmai-NL]

@deepu105 Does @Amanieu's explanation suffice for you? If so, can you close the issue?