Cannot build for mips and armhf with jemalloc

[herrernst]

I tried cross-compiling to mips, getting jemalloc errors when linking:

rustc hello.rs --target=mips-unknown-linux-gnu -C linker=mips-openwrt-linux-gcc
'generic' is not a recognized processor for this target (ignoring processor)
...
'generic' is not a recognized processor for this target (ignoring processor)
error: linking with `mips-openwrt-linux-gcc` failed: exit code: 1
note: "mips-openwrt-linux-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:843: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:1146: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_insert':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_guard'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/arena.c:91: undefined reference to `__stack_chk_fail'
/home/user/.multirust/toolchains/nightly-2016-01-01/lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib(arena.pic.o): In function `arena_run_tree_remove':

... and it goes on.
When adding this to main.rs it works

#![feature(alloc_system)]
extern crate alloc_system;

Error happens with stable 1.5, beta, latest nighty. I'm using the linker from OpenWRT SDK which identifies as "mips-openwrt-linux-gcc (OpenWrt/Linaro GCC 4.8-2014.04 r46450) 4.8.3"

Also I get a similar error when trying to compile for armhf:

rustc hello.rs --target=arm-unknown-linux-gnueabihf -C linker=~/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc
error: linking with `/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc` failed: exit code: 1
note: "/home/user/src/pi-tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian-x64/bin/arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "hello.0.o" "-o" "hello" "-Wl,--gc-sections" "-pie" "-nodefaultlibs" "-L" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib" "-Wl,-Bstatic" "-Wl,-Bdynamic" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libstd-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcollections-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librustc_unicode-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/librand-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liblibc-17a8ccbd.rlib" "/home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/libcore-17a8ccbd.rlib" "-l" "dl" "-l" "pthread" "-l" "gcc_s" "-l" "pthread" "-l" "c" "-l" "m" "-l" "rt" "-l" "compiler-rt"
note: /home/user/.multirust/toolchains/nightly-2016-01-15/lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib(jemalloc.pic.o): In function `malloc_conf_init':
/buildslave/rust-buildbot/slave/nightly-dist-rustc-cross-linux/build/src/jemalloc/src/jemalloc.c:919: undefined reference to `secure_getenv'
collect2: error: ld returned 1 exit status

error: aborting due to previous error

Here, I am using gcc from Raspberry Pi tools.

[MagaTailor]

Looks like a simple case of missing symbols - probably means the rlibs you're trying to use were built and linked against incompatible libraries.

EDIT:
You mention rpi - that immediately incriminates old glibc.

[herrernst]

I've downloaded the rust libs from http://static.rust-lang.org/dist/ndex.html, e.g. http://static.rust-lang.org/dist/2016-01-15/rust-std-nightly-mips-unknown-linux-gnu.tar.gz

find . -name "*jemalloc*"
./lib/rustlib/mips-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib
./lib/rustlib/arm-unknown-linux-gnueabihf/lib/liballoc_jemalloc-17a8ccbd.rlib
./lib/rustlib/x86_64-unknown-linux-gnu/lib/liballoc_jemalloc-17a8ccbd.rlib

Is this the wrong approach? Should I build them myself?

[MagaTailor]

Yes, cause you're using a slightly old distro. If you tried it on one of the Odroids that would work as expected.

[alexcrichton]

Interesting! This is likely because we're using a pretty new toolchain to compile the MIPS nightlies, and we may need to downgrade that. This may be the same case for the ARM nightlies as well.

[lilianmoraru]

@alexcrichton Or just provide more options?
Edit:
Actually, yes, downgrading might be a better option.

[MagaTailor]

@herrernst

If you try to provide your own rlibs just be careful to not bump into this issue, relevant comment: #30363 (comment)

[herrernst]

Thanks. I'm quite new to rust. Cross-compiling std doesn't seem to be straightforward, so I'm just using the system allocator for now.
Update: I've now compiled Rust with mips target, and then using jemalloc also works.

[lilianmoraru]

@herrernst So, can the guys close this issue?
Who is interested, here is more on this issue.

[herrernst]

I think it should be considered downgrading the build chain for the prebuilt std libs, as alexcrichton has said, because it would be nice for example to use the standard Raspberry Pi toolchain without having to need compiling Rust yourself.

[joerg-krause]

@herrernst This is my approach https://github.com/joerg-krause/rust-cross-libs to cross-compile std for an ARMv5 target (which is an custom target), because I did not wanted to cross-bootstrap the whole Rust toolchain.

EDIT: insert link

[alexcrichton]

I would indeed like to assuage this issue if at all possible, but I also want to try to drill down here and get some specifics about how to move forward. What I believe to be the problem is that we're using "too new" of a C compiler when compiling jemalloc, and I suspect that this may be fixable by just using an older C compiler. With that in mind, first here's what we're using on the bots for building our nightlies:

• Ubuntu 15.10
• The ARM compiler is the gcc-arm-linux-gnueabi package with the version 4:5.2.1-1
• The MIPS compiler is the gcc-5-mips-linux-gnu package with the version 5.2.1-22ubuntu2mips1~ubuntu15.10.1cross1

Both of those gcc versions are indeed very new, but I wouldn't be 100% sure of how to downgrade, either, unfortunately. I suspect what actually needs to be downgraded is the glibc version for each platform (it just tends to be in step with a gcc version). With this all in mind, I think there's a few courses of action here:

1. Figure out if there's a "standard toolchain" here that we can use. Can we pick some older version of gcc/glibc which works for almost everyone, get some compilers for that, and use those to build nightlies?
2. Disable jemalloc by default on these platforms for now. This is, I believe, the only C code left in the standard library by default, so we should in theory erase that level of glibc portability issues if we stop including any C code.
3. Document what toolchains we're compiling jemalloc with and say that they're the minimum required version.

I'd personally be in favor of 1 or 2, but I unfortunately don't know what to do with option 1...

---

As a side note as well, it looks like the OpenWRT SDK uses uclibc? Does that strive for binary ABI compatibility with glibc? If not then there may be worse consequences mixing with the mips-unknown-linux-gnu target if the standard library doesn't agree with the uclibc library about various constant values and such. To solve that problem we could provide a mips-unknown-linux-uclibc, however.

[lilianmoraru]

On Ubuntu 15.10 the oldest GCC that can be installed is GCC 4.7.
Usually, if there is a package for GCC [version], than there are also available cross-compilers with the same versions.
So, instead of installing gcc-arm-linux-gnueabi it should just be changed into gcc-4.7-arm-linux-gnueabi and instead of gcc-arm-linux-gnueabihf -> gcc-4.7-arm-linux-gnueabihf.
GCC will pick the correct glibc on its own, nothing else to do here.
Note that the binary is actually called arm-linux-gnueabi-gcc-[version] and care has to be taken that the link arm-linux-gnueabi-gcc points to the correct version(It could point to the wrong version if another version was installed before this), in this case 4.7.

To the second question:
It should not be ok to use gcc that uses glibc to compile for a target that actually has all its libraries compiled against uclibc. The API might look the same and the software might compile but it is enough to have 1 more variable or the variables arrangement to be different and there will be some crazy bugs because of the memory layout.

Here is a comparison - you can see that uclibc doesn't even have a stable ABI.

ref Document what toolchains we're compiling jemalloc with and say that they're the minimum required version. - The archive name could probably include the compiler that was used, the same way it indicates the target triple.

[alexcrichton]

Thanks for the answer @lilianmoraru! The pointer for the gcc-4.7-arm-linux-gnueabihf package looks like it works for me, so all we need to do is test out it still compiles a toolchain ok and then I'd be fine updating the image.

For MIPS we have a custom PPA from @angelsl, so unfortunately there isn't a standard package for 4.7 (I believe) just yet. We can likely try to find one, however, or make it ourselves.

Also thanks for explaining uclibc, that matches what I'd suspect. I'm a little surprised that it doesn't even have a stable ABI (could make, e.g., porting liblibc to it a little difficult), but alas! In that case anyway @herrernst I don't think you can use our MIPS nightlies with the OpenWRT toolchain because there's a mismatch with the libc versions.

[herrernst]

Thanks for all the information and feedback work. One thing I'm not yet quite sure of: If a build a local std crate with the compiler from the OpenWRT toolchain, should it work then? It then links against uclibc, doesn't it? I picked some random Rust project, hyper, and the example server (albeit without OpenSSL for now) seems to run okay.

[lilianmoraru]

@herrernst, seems to run okay on the OpenWRT target? If yes, my guess would be that everything you need is statically linked(like it would be with musl). Probably you don't use system libraries(dynamic linking).

You could check that by running ldd on your binary. You can find the correct ldd in your OpenWRT SDK.

[japaric]

@alexcrichton

As a side note as well, it looks like the OpenWRT SDK uses uclibc?

The trunk version of OpenWRT is now using musl instead of uclibc. The next stable release will probably be using musl, so having mips-unknown-linux-musl nightlies would be nice.

@herrernst

Apart from building std against uclibc, I think we'll also need to add uclibc support to the libc crate.

[alexcrichton]

Whether libstd works with uclibc may actually be either intended or lucky. When compiling the standard library for MIPS on Linux it will default to glibc definitions of libc (e.g. this is what liblibc is verified against), so if it works against uclibc then either:

A) uclibc tries to mirror glibc in terms of ABI, but doesn't provide guarantees and we just happen to use the subset that mirrors correctly
B) uclibc just happens to align with glibc in the subset of uclibc we use.

Overall I would say that it's pretty risky business to run a -gnu nightly against a uclibc build

---

@japaric

oh excellent! I'd be quite interested in getting those builds up and running.

[lilianmoraru]

@alexcrichton uclibc does not attempt to keep ABI compatibility with glibc:
It is[uClibc] also highly configurable in supported features, at the cost of ABI differences for different configurations.

My only guess would be that it works because it was statically linked.
The same way musl works. That's the main advantage of musl - you compile against it and then you can run the binary even on very old Linux systems, because you don't use the system libraries.

[herrernst]

I don't think it's statically linked:

rustc --target=mips-unknown-linux-gnu  hello.rs -C linker=mips-openwrt-linux-uclibc-gcc
objdump -p hello | grep NEEDED
  NEEDED               libdl.so.0
  NEEDED               libpthread.so.0
  NEEDED               libgcc_s.so.1
  NEEDED               libc.so.0
  NEEDED               ld-uClibc.so.0
  NEEDED               libm.so.0

On the target:

root@OpenWRT:~# LD_TRACE_LOADED_OBJECTS=1 ./hello 
    libdl.so.0 => /lib/libdl.so.0 (0x777b4000)
    libpthread.so.0 => /lib/libpthread.so.0 (0x7778e000)
    libgcc_s.so.1 => /lib/libgcc_s.so.1 (0x7776a000)
    libc.so.0 => /lib/libc.so.0 (0x776fd000)
    ld-uClibc.so.0 => /lib/ld-uClibc.so.0 (0x777c8000)
    libm.so.0 => /lib/libm.so.0 (0x776d7000)

I think the hello-world example works because "uClibc is more compatible to
glibc [than musl], so most of the software packages used on embedded systems are
working fine without patching".

[japaric]

Triage: still getting the "undefined reference to secure_getenv" on ARM with Rust 1.7.0 and nightly-2016-03-07 when cross compiling hello world. Oddly enough, I didn't get this error when cross compiling with Rust 1.6.0 and 1.8.0-beta.1

[alexcrichton]

Hm interesting, @japaric looks like there's a gcc version mismatch. We're 4.7 on the bots, and it looks like you're using 4.6.3?

Not entirely sure why the other two versions would work...

[japaric]

@alexcrichton I think the problem is libc and not gcc. secure_getenv was introduced in glibc-2.17 (grep for secure_getenv). The bots are running Ubuntu 15.10 so the libc6-dev-armel-cross package is glibc-2.19 glibc-2.21. Travis is running precise, the libc6-dev-armel-cross package there is glibc-2.15.

A solution would be moving the bots to an older arm glibc. crosstool-ng can generate a gcc-4.8/glibc-2.14 arm toolchain, perhaps we could use that in the bots?

EDIT: Actually the bots are running ubuntu 15.10, so the glibc version is higher.

[alexcrichton]

Ah yeah that'd do it. I tend to conflate gcc/glibc as they're so commonly paired together. I'd love to use an older glibc!

If you've got a way to updated our linux-cross image then I'd be down to merge+deploy.

[japaric]

Triage: the problem is gone on ARM, today's (2016-03-16) libstd can be used to cross compile to arm-unknown-linux-gnueabihf on Ubuntu precise.

[jan-hudec]

Confirming, nightly does not have the problem. Stable does.

I am then hitting another problem where qemu complainins it does not support system call 384 when trying to run the tests. Somehow it only complains in some builds and I don't see any obvious difference between those that worked and those that didn't.

[jan-hudec]

Note that the other option is to request Trusty from TravisCI. I tested that the settings to request it, sudo: required and dist: trusty can be placed under matrix.include and thus only used for the arm cross-compiles that don't work otherwise.

[alexcrichton]

I believe this has since been fixed, so closing.

[jan-hudec]

Yes, for nightly it compiles now.

The tests unfortunately don't run on Travis under qemu anyway, neither in the default container nor with dist: trusty. Most of the time they just print message tcg fatal error (with various source line) and lock up.