link: failed to static link to c++ library when global variable is used

[BusyJay]

Hi, I am trying to build a binary written in rust static linked with a c++ library. I follow the instructions in official documentation, but end up with an error：

undefined reference to `__dso_handle'

Following is a simple example:

// test.cpp
#include <vector>

const std::vector<int> ids = {2, 3, 4};

extern "C" {
    extern int get_id(int idx);
}

int get_id(int idx) {
    return ids[idx];
}

// main.rs
#[link(name = "test")]
extern "C" {
    pub fn get_id(idx: usize) -> usize;
}

fn main() {
    unsafe {
        println!("id at {} is {}", 2, get_id(2));
    }
}

I build the static library using following command:

g++ -std=c++0x -c test.cpp -o test.o
ar rcs libtest.a test.o

And build the binary using following command:

LIBRARY_PATH=./ rustc main.rs -lstdc++ --target=x86_64-unknown-linux-musl

It failed with error:

note: ./libtest.a(test.o): In function __static_initialization_and_destruction_0(int, int)': test.cpp:(.text+0xa2): undefined reference to__dso_handle'
/usr/local/lib/rustlib/x86_64-unknown-linux-musl/lib/libstd-8102e29f.rlib(std-8102e29f.0.o):(.data._rust_extern_with_linkage___dso_handle+0x0): undefined reference to `__dso_handle'

I can build the binary successfully when linking dynamically. Am I missing any thing here? Any help is appreciated.

[nagisa]

You are required to define two functions and one symbol in order to use global objects in your C++ files: __dso_handle symbol, __cxa_atexit and __cxa_finalize functions. These are part of Itanium C++ ABI and thus rustc has no responsibility of providing any of the symbols or calling any of the functions.

---

As an aside, that C++ snippet probably does not do what you expect it to do.

[BusyJay]

I don't get it, does it mean I can't static link to libraries that is written in C++ and uses global variables? Or is there an extra library that I should link to?

This is just a simple example to reproduce the failure. And it actually works.

[nagisa]

does it mean I can't static link to libraries that is written in C++ and uses global variables

Not without extra effort on your part, no. Remember that you’re mixing the GNU C runtime (which defines the __dso_handle in an object file somewhere at /usr/lib/gcc/<target>/<version>/crtbegin.o) and MUSL runtime which probably has its own internal symbols and stuff to do that.

Or is there an extra library that I should link to?

Anything related to linking more libraries would likely not work as it would introduce symbol conflict between runtimes. Instead, either build your C++ object targetting MUSL, or use the -gnu target on the rust side.

[alexcrichton]

Ah yes indeed, if g++ is a GNU gcc then linking those object files into a Rust musl program won't work (as you're seeing with weird symbol errors), unfortunately.

[Mark-Simulacrum]

From what I can tell, this isn't a bug in rustc. Closing.

[bossmc]

After keeping my CPU warm for a while by compiling:

• musl
• binutils
• libunwind
• g++ (targeting musl)

I now have in my hands a musl-compatible crtbegin(T/S).o and crtend(S?).o and, by compiling a modified rustc (only to stage2 thankfully 😥) and making the link command be:

cc .. crt1.o crti.o crtbeginT.o ... crtend.o crtn.o

I am able to create a static binary containing Rust and C++ code (in my case I was trying to compile the example code from https://github.com/tupshin/cassandra-rs which uses a C++ driver). So, this is possible 🎉 🎈!

Now, not everyone is going to go to the trouble I did above, but there are good use cases for static binaries (for one, they actually run everywhere, unlike "mostly static" binaries that are still glibc version dependent), and it would be nice to be able to modify the "early objects" and "late objects" lists that get passed to the linker.

#![link(name=crtbegin.o)] or cargo:rustc-link-lib=static=crtbeginT.o aren't sufficient since they don't control the order the extra libraries are added to the link command (also I'm not sure they work for .o files but I could easily make a libcrtbeginT.a if this was going to work).

[dl00]

@bossmc is it possible you could share this modified rustc? I am currently working on a cross compiling toolchain and I have the crt object files you mention... but of course rust fails on a C++ library.

[bossmc]

Not from this PC, but the only change I made (I think 🤞) was to modify https://github.com/rust-lang/rust/blob/master/src/librustc_back/target/linux_musl_base.rs#L59-L61 to add the crtBeginT.o and crtEnd.o at the appropriate points (and to copy those files to the same path rustc installs it's copy of crti.o etc).

There are a few alternatives I've considered since:

• Rust brings it's own C++ runtime objects (as it does with C runtime on musl)
  • Seems dangerous, I doubt libc++ from GNU and from LLVM have compatible run-time objects
• Rust allows the linker to add the run-time objects (as it does with the -gnu target)
  • I'm not sure it's possible to ask the linker to add the run-time objects and not add the rest of the standard libraries
  • It's not clear how to work out whether to include the C++ run-time, maybe include it unilaterally?
• Add config flags to rustc (and Cargo's config - https://doc.rust-lang.org/cargo/reference/config.html#configuration-keys) to specify extra run-time objects to be included in the link commands
  • Seems like a decent solution - given that I don't think rustc can assume the provenance of the libc++.a like it can with the musl libc.a (though even that is a little dubious in my mind)
• Check if simply creating libcrt++.a from crtBeginT.o and crtEnd.o then adding cargo:rustc-link-lib=static=crt++ is sufficient
  • If this works it's a pretty good workaround, but doesn't feel like a good long-term solution (what if you link in two crates that both link to C++ libraries... you only want the runtime objects once)
• Use a custom linker wrapper script which adds these objects to the objects rustc asks for
  • Sensible for a cross compiler, not so nice for a native compiler

@alexcrichton - I'd quite like to help resolve this, who should I be working with/talking to? Do we need an RFC?

[alexcrichton]

@bossmc er sorry this is pretty out of cache for me, but if you've got some linking questions or whatnot feel free to hit me up on IRC!

[bossmc]

Fair enough, you were the last name I recognised as a core dev to comment. I'll shout in IRC at some point when I've got some time.

[dl00]

For whom it may concern, I used the last suggestion @bossmc of creating a custom linker script. After some troubleshooting, this is what I came up with that works:

#!/bin/bash

args=()

for arg in "$@"; do
	if [[ $arg = *"Bdynamic"* ]]; then
		args+=() # we do not want this arg
	elif [[ $arg = *"crti.o"* ]]; then
		args+=("$arg" "/usr/arm-unknown-linux-musleabi/lib/gcc/arm-unknown-linux-musleabi/6.3.0/crtbeginT.o" "-Bstatic")
	elif [[ $arg = *"crtn.o"* ]]; then
		args+=("-lgcc" "-lgcc_eh" "-lc" "/usr/arm-unknown-linux-musleabi/lib/gcc/arm-unknown-linux-musleabi/6.3.0/crtend.o" "$arg")
	else
		args+=("$arg")
	fi
done
#echo "RUNNING WITH ARGS: ${args[@]}"
arm-unknown-linux-musleabi-real-g++ "${args[@]}"

The first if removes Bdynamic args, which is for some reason still included in static linking. Then, it adds crtbegin.o after it sees crti.o, and similar for crtn.o. For some reason a bunch of compiler symbols were still missing after I added those options, so I also added -lgcc -lgcc_eh -lc to ensure other libraries that were missing were included.

It might be a bit sloppy, but that is what worked for me.

[bossmc]

Wild speculation, but were the missing symbols related to _Unwind_Resume and similar? If so, you might consider using LLVM's libunwind, which is what the rustc build uses on musl platforms.

You'll need libc since libc++ depends on it, but you shouldn't (need to) be dependent on any gcc libraries. You could also link directly to libc.a rather than using -lc if you prefer.

[dl00]

Actually, the missing symbols were like __sync_fetch_and_add_8 which required -lgcc for sure. TBH, I do not even know if I needed -lc, but I noticed it was always included with -lgcc so I figured I would just include it because I was tired of waiting an hour too see if it would succeed :)

[bossmc]

Did you add <atomic.h> support to your musl build? I used http://stdatomic.gforge.inria.fr/ to add self-contained atomics (rather than depend on GCC's ones).

Yeah, I know the feeling about the multiple hour wait for bootstrap compilations 😀...

[dl00]

hmm... that the library you proposed does look like a good alternative especially considering that that was basically all I was missing after fixing crtbegin/crtend issues. I think I will give it a try within the next weeks--its a great suggestion.

[jonas-schievink]

Reopening as per #36710 (comment) (feel free to ping me or another team member if bugs need to be reopened!)

[jonas-schievink]

This seems to work fine now, as long as -C link-self-contained=no is used (or autodetected), so I've opened #77901 to reenable the test.

[bossmc]

@jonas-schievink I think that means that a self-contained link will still not work for a C++ binary (even if the C++ bit is built with musl libc)? Is that just inevitable?

[jonas-schievink]

For now, yes, that still won't work.

[mati865]

Is that just inevitable?

CRT objects are not guaranteed to be compatible between different compilers, so the answer is yes.