Getting a Illegal instruction segfault on arm (Raspberry Pi 1)

[weiznich]

Originally reported as tokio-rs/tokio-core#324

I'm trying to run some future related code on an old raspberry pi 1. After cross compiling a binary construction a reactor and spawning a future on it the program segfaults with an illegal hardware instruction error.
Programs without a reactor are running fine.

Example code:

extern crate tokio_core;
extern crate futures;

fn main() {
   let core = tokio_core::reactor::Core::new().unwrap();
       core.handle().spawn(futures::future::ok(()));
    println!("Hello, world!");
}

Versions:

• Rustc: rustc 1.25.0 (84203cac6 2018-03-25), rustc 1.27.0-nightly (79252ff4e 2018-04-29)
• tokio_core: 0.1.17
• futures: 0.1.21
• target: arm-unknown-linux-musleabihf

[estebank]

@weiznich could you verify if this ever worked on previous versions of the compiler?

[mattico]

Could you collect a stacktrace? Something like this should work:

$ ulimit -c unlimited # enable core dumps
$ cargo run
$ gdb ./target/debug/PROGRAM core.PROGRAM.PID
(gdb) bt

[weiznich]

Could you collect a stacktrace?

(I've just run the test program in gdb)

(gdb) bt
#0  printf_core (f=f@entry=0x0, fmt=fmt@entry=0x11a1d1 "Symbol not found: %s", ap=ap@entry=0x7effe6ec, nl_arg=nl_arg@entry=0x7effe718, nl_type=nl_type@entry=0x7effe6f0) at src/stdio/vfprintf.c:618
#1  0x0010179c in vfprintf (f=f@entry=0x7effe7f0, fmt=fmt@entry=0x11a1d1 "Symbol not found: %s", ap=ap@entry=...) at src/stdio/vfprintf.c:666
#2  0x001019e8 in vsnprintf (s=s@entry=0x0, n=n@entry=0, fmt=fmt@entry=0x11a1d1 "Symbol not found: %s", ap=...) at src/stdio/vsnprintf.c:54
#3  0x00105894 in __dl_vseterr (fmt=0x11a1d1 "Symbol not found: %s", fmt@entry=0x7efff55c "<\365\377~\354\365\377~\344\365\377~\001", ap=..., ap@entry=...) at src/ldso/dlerror.c:34
#4  0x001058ec in __dl_seterr (fmt=0x11a1d1 "Symbol not found: %s") at src/ldso/dlerror.c:51
#5  0x0010581c in stub_dlsym (p=<optimized out>, s=<optimized out>, ra=<optimized out>) at src/ldso/__dlsym.c:9
#6  0x000c8dc4 in mio::sys::unix::dlsym::fetch (name=...) at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/mio-0.6.14/src/sys/unix/dlsym.rs:43
#7  0x000c8b60 in <mio::sys::unix::dlsym::DlSym<F>>::get (self=0x14506c <mio::sys::unix::epoll::Selector::new::epoll_create1>) at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/mio-0.6.14/src/sys/unix/dlsym.rs:30
#8  0x000c8e7c in mio::sys::unix::epoll::Selector::new () at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/mio-0.6.14/src/sys/unix/epoll.rs:38
#9  0x000bd148 in mio::poll::Poll::new () at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/mio-0.6.14/src/poll.rs:655
#10 0x000a934c in tokio_reactor::Reactor::new () at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/tokio-reactor-0.1.1/src/lib.rs:215
#11 0x00053b78 in tokio::runtime::builder::Builder::build (self=0x7effeff0) at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/tokio-0.1.5/src/runtime/builder.rs:94
#12 0x00058ccc in tokio::runtime::Runtime::new () at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/tokio-0.1.5/src/runtime/mod.rs:260
#13 0x0004c72c in tokio_core::reactor::Core::new () at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/tokio-core-0.1.17/src/reactor/mod.rs:129
#14 0x000129a8 in tokio_segfault_test::main () at src/main.rs:5

could you verify if this ever worked on previous versions of the compiler?

The oldest version that compiles the test snipped is rust 1.21. (Due to some features in crossbeam). It is already broken there.

---

Compiling the same code using the arm-unknown-linux-gnueabihf target produces a segfault due to a illegal instruction.

(gdb) bt
#0  0x00408602 in <crossbeam_utils::cache_padded::CachePadded<T> as core::ops::deref::DerefMut>::deref_mut (
    self=0x7efff5b4)
    at /root/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/crossbeam-utils-0.3.2/src/cache_padded.rs:147
Backtrace stopped: Cannot access memory at address 0x4

Using the arm-unknown-linux-gnueabi target produces a segmentation fault due to some memory issue 😟

(gdb) bt
#0  0x76ddfdd8 in _dl_update_slotinfo () from /usr/lib/ld-linux.so.3
#1  0x00000160 in ?? ()
Backtrace stopped: previous frame identical to this frame (corrupt stack?)

Using the arm-unknown-linux-musleabi target produces a binary that runs without segfault.

[mattico]

Some raspi distros use the soft-float ABI, apparently, which could explain the gnueabihf issue.

Not sure about the second issue, maybe a glibc versioning issue? Can you try compiling on the raspi?

[weiznich]

Some raspi distros use the soft-float ABI, apparently , which could explain the gnueabihf issue.

According to that link Arch Linux (the distro running on the Pi) supports the *hf target since 2012. Also a program without a tokio reactor works just fine. I assume this won't be the case if I've chosen the wrong target?

Can you try compiling on the raspi?

I will try that later.

[weiznich]

Can you try compiling on the raspi?

Building the code above on a pi using the arm-unknown-linux-gnueabihf target produces a working binary.

[parched]

Looks like the illegal instruction is in musl, where did you get musl from, maybe it's build for Armv7? Can you print the actual instruction in gdb that causes the problem?

[weiznich]

Looks like the illegal instruction is in musl, where did you get musl from, maybe it's build for Armv7?

All targets where installed via rustup. For building against musl I used this docker file as base (I've used the arm-musleabihf variant.)

Can you print the actual instruction in gdb that causes the problem?

   │0xffff8 <printf_core>           push   {r4, r5, r6, r7, r8, r9, r10, r11, lr}
   │0xffffc <printf_core+4>         vpush  {d8-d14} 
  >│0x100000 <printf_core+8>        vmov.f64       d10, #112       ; 0x3f800000  1.0    
   │0x100004 <printf_core+12>       vmov.f64       d11, #120       ; 0x3fc00000  1.5      
   │0x100008 <printf_core+16>       vldr   d8, [pc, #904]  ; 0x100398 <printf_core+928>   
   │0x10000c <printf_core+20>       vldr   d9, [pc, #908]  ; 0x1003a0 <printf_core+936>

[parched]

That's a VFPv3 instruction (raspi 1 has VFPv2), so it looks like musl has been compiled with the wrong flags.

[weiznich]

That's a VFPv3 instruction (raspi 1 has VFPv2), so it looks like musl has been compiled with the wrong flags.

So I've tried to rebuild the whole musl toolchain from scratch using the same flags as the gcc installed on the raspberry pi, but I'm getting the same Illegal Instruction error as before.

gcc on pi:

$ gcc -v
gcc -v
Using built-in specs.
COLLECT_GCC=gcc
COLLECT_LTO_WRAPPER=/usr/lib/gcc/armv6l-unknown-linux-gnueabihf/7.3.1/lto-wrapper
Target: armv6l-unknown-linux-gnueabihf
Configured with: /build/gcc/src/gcc/configure --prefix=/usr --libdir=/usr/lib --libexecdir=/usr/lib --mandir=/usr/share/man --infodir=/usr/share/info --with-bugurl=https://github.com/archlinuxarm/PKGBUILDs/issues --enable-languages=c,c++,fortran,go,lto,objc,obj-c++ --enable-shared --enable-threads=posix --with-system-zlib --with-isl --enable-__cxa_atexit --disable-libunwind-exceptions --enable-clocale=gnu --disable-libstdcxx-pch --disable-libssp --enable-gnu-unique-object --enable-linker-build-id --enable-lto --enable-plugin --enable-install-libiberty --with-linker-hash-style=gnu --enable-gnu-indirect-function --disable-multilib --disable-werror --enable-checking=release --enable-default-pie --enable-default-ssp --host=armv6l-unknown-linux-gnueabihf --build=armv6l-unknown-linux-gnueabihf --with-arch=armv6 --with-float=hard --with-fpu=vfp
Thread model: posix
gcc version 7.3.1 20180406 (GCC)

(musl)gcc in docker container

$ gcc -v
Using built-in specs.
COLLECT_GCC=arm-unknown-linux-musleabihf-gcc
COLLECT_LTO_WRAPPER=/usr/local/musl/bin/../libexec/gcc/arm-unknown-linux-musleabihf/7.2.0/lto-wrapper
Target: arm-unknown-linux-musleabihf
Configured with: ../src_gcc/configure --enable-languages=c,c++ --with-float=hard CFLAGS='-g0 -Os' CXXFLAGS='-g0 -Os' LDFLAGS=-s --disable-nls --with-debug-prefix-map=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf= --enable-languages=c,c++ --disable-libquadmath --disable-decimal-float --disable-multilib --with-arch=armv6 --with-float=hard --with-fpu=vfp --disable-werror --target=arm-unknown-linux-musleabihf --prefix= --libdir=/lib --disable-multilib --with-sysroot=/arm-unknown-linux-musleabihf --enable-tls --disable-libmudflap --disable-libsanitizer --disable-gnu-indirect-function --disable-libmpx --enable-deterministic-archives --enable-libstdcxx-time --with-build-sysroot=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_sysroot AR_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/ar AS_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/gas/as-new LD_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/ld/ld-new NM_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/nm-new OBJCOPY_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/objcopy OBJDUMP_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/objdump RANLIB_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/ranlib READELF_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/readelf STRIP_FOR_TARGET=/tmp/musl-cross-make/build/local/arm-unknown-linux-musleabihf/obj_binutils/binutils/strip-new
Thread model: posix
gcc version 7.2.0 (GCC

I'm not sure if I missed something but I thing I set all relevant flags to be the same (--with-arch=armv6 --with-float=hard --with-fpu=vfp )

[mattico]

Did you also recompile musl libc with that compiler?

[weiznich]

I think so. The docker file uses this to build the musl cross compiler. I've build everything from scratch.

[mattico]

Wow this is a real cross-compile hell.

Rustc might be using a different gcc. Try cargo rustc --target=arm-unknown-linux-musleabihf -- -C linker=gcc.

Maybe rustc/gcc is finding libc in the wrong place? We can find out by looking at the output of ld --verbose. There's probably an easier way to do this but this is what I did:

# note that I'm using gnueabihf on ubuntu so the commands are a bit different
$ cargo +nightly -v rustc --target=arm-unknown-linux-gnueabihf -- -C linker=arm-linux-gnueabihf-gcc -Z print-link-args
   Compiling test-cross v0.1.0 (file:///home/matt/test-cross)
     Running `rustc --crate-name test_cross <snip>
# Paste the rustc invocation here so we get the print-link-args output
$ rustc --crate-name test_cross  <snip>
"arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" <snip>
# paste the gcc invocation here, and add `-v -Wl,--verbose`
$ "arm-linux-gnueabihf-gcc" "-Wl,--as-needed" "-Wl,-z,noexecstack" <snip> -v -Wl,--verbose
<lots of output>

In the ld --verbose output it should say where it's finding libc.a. If that's not the libc.a from your musl toolchain, then that's your problem.

[bmisiak]

FWIW, I'm experiencing a similar issue with tokio. Tried both musl cross-compilation from macOS (using the lld linker in nightly) and gnueabihf compilation, from Ubuntu 18.04 on WSL, using arm-linux-gnueabihf-gcc-8 as the linker.

[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/arm-linux-gnueabihf/libthread_db.so.1".

Program received signal SIGILL, Illegal instruction.
0x00407256 in _$LT$core..result..Result$LT$T$C$$u20$E$GT$$GT$::unwrap::hb50baa32cb0f81c2 (self=<error reading variable: Cannot access memory at address 0x1>)
    at /checkout/src/libcore/result.rs:779
779	/checkout/src/libcore/result.rs: No such file or directory.

[weiznich]

Maybe rustc/gcc is finding libc in the wrong place? We can find out by looking at the output of ld --verbose.

So I've dug into this a bit further. If I'm not misreading the output of ld --verbose the linked libc.a lives in the rustup installation dir and seems to be part of the toolchain.

(I've attach the full log, so somebody else could verify my conclusions. The relevant part seems to be around line 700)

If someone has further ideas to try I'm happy to continue debugging this 😉

[mattico]

Aha, of course libc is shipped with the compiler, I just didn't know if the libc.rlib actually contained libc.a or not. I suppose for musl it would have to. This looks like its a bug in rustc or libc or cc. The arm-unknown-linux-musleabihf target (and friends) are supposed to use vfp2, but musl is getting compiled with vfp3 enabled for some reason.

[mattico]

I checked, and neither cc-rs nor the dist-builder's ct-ng script specify the -mfpu option for these targets.

However, from what I've seen no ARMv6 processor supports VFP3 instructions, so they should never get emitted by default. The prinf_core implementation doesn't use any assembly code from what I can tell, so somehow the C is emittin VFP3 instructions. Maybe the compiler could be convinced to produce them if the -mfpu=vfp3 flag is added, even on armv6?

Edit: it seems that instruction requires vfp3 only for immediate arguments. Maybe there's a bug in GCC where it doesn't know that information?

[weiznich]

Is there anything that I can to help fixing this?

[mattico]

I managed to build musl without the offending instruction using this configure invocation:

CC=arm-linux-gnueabihf-gcc CFLAGS="-marm -mfpu=vfp -mfloat-abi=hard -mcpu=arm1176jzf-s" ./configure --prefix=$PWD/musl-arm-gcc-vfp --enable-wrapper=yes

So you'll want to figure out a way to get these gcc flags (-marm -mfpu=vfp -mfloat-abi=hard -mcpu=arm1176jzf-s) into your docker container.

[piersfinlayson]

I've seen this with pretty much the same symptoms as @weiznich. A workaround appears to be to src/liblibc/ci/docker/arm-unknown-linux-musleabihf/Dockerfile, adding -mfpu=vfp to the CFLAGS:

RUN CC=arm-linux-gnueabihf-gcc \
    CFLAGS="-march=armv6 -marm -mfpu=vfp" \
    ./configure --prefix=/musl-arm --enable-wrapper=yes

I've built musl libc.a both with and without -mfpu=vfp, and without the -mfpu flag I get a binary with the illegal variant of vmov.f64, and with the flag I don't get it.

This actually looks like a GCC bug to me - as @parched says above, it looks like VFPv3 just isn't supported on ARMv6 - this ARM doc backs that statement up.

If you run:

arm-linux-gnueabihf-gcc -march=armv6 -marm -Q --help=target

then you get:

<snip>
-mfpu=                                vfpv3-d16
<snip>

But if you run:

arm-linux-gnueabihf-gcc -march=armv6 -marm -mfpu=vfp -Q --help=target

you get:

-mfpu=                                vfp

I also get the wrong answer if I put just -mcpu=arm1176jzf-s, so it's the -mfpu=vfp to force VFPv2 which is doing the trick.

[mati865]

#56725 was merged, is this still an issue?

[piersfinlayson]

#56725 was merged, is this still an issue?

The fix isn't in 1.31.1. I'm hoping it'll be in 1.32.

[prisme60]

Tested OK on Rust 1.33 on my Rasberry Pi 1 on Alpine distribution (musl armhf) with cross-compilation.
Previous versions of Rust were not working.

[piersfinlayson]

Agree, this is fixed as of Rust 1.33