Many tests failing with "Couldn't compile the test" on 32-bit PowerPC with 1.49 beta

[glaubitz]

Since 1.49 beta, many tests fail on 32-bit PowerPC with the error that the test could not be compiled:

running 1 test
test src/test/rustdoc/async-move-doctest.rs - (line 8) ... FAILED

failures:

---- src/test/rustdoc/async-move-doctest.rs - (line 8) stdout ----
Couldn't compile the test.

failures:
    src/test/rustdoc/async-move-doctest.rs - (line 8)

test result: FAILED. 0 passed; 1 failed; 0 ignored; 0 measured; 0 filtered out

I have not bisected the issue yet.

Full log in: https://buildd.debian.org/status/fetch.php?pkg=rustc&arch=powerpc&ver=1.49.0%7Ebeta.4%2Bdfsg1-1%7Eexp1&stamp=1611482142&raw=0

CC @jrtc27 @infinity0 @smaeul @mator

[bjorn3]

I am seeing a lot of signal 4 (SIGILL) errors in the build log.

[glaubitz]

Ah, that reminds me of one bug in LLVM: https://bugs.llvm.org/show_bug.cgi?id=46683

I'll check whether this one fixes it: llvm/llvm-project@a5d161c

[glaubitz]

Debian's LLVM 11 package (llvm-toolchain-11) already contains the fix, so it can't be this.

I'm trying to bisect this now in the Rust compiler itself but I just stumbled into the problem that a cross build no longer builds a full stage2 compiler with the std compiler which can be used on the target system, so I will first have to figure out what changed in Rust here.

[bjorn3]

You can pass --stage 2 to ./x.py. The default changed to --stage 1.

[glaubitz]

You can pass --stage 2 to ./x.py. The default changed to --stage 1.

Thanks, that's what I guessed already, I just couldn't find the change in the vast amount of other changes ;).

[bjorn3]

https://blog.rust-lang.org/inside-rust/2020/08/30/changes-to-x-py-defaults.html: #73964

[glaubitz]

Just cross-compiled 1.50 and the issue is still there:

(sid_powerpc-dchroot)glaubitz@perotto:~/rustc/rustc-1.50.0+dfsg1$ CARGO_PKG_HOMEPAGE='https://github.com/alexcrichton/cfg-if' CARGO_PKG_LICENSE=MIT/Apache-2.0 CARGO_PKG_LICENSE_FILE='' CARGO_PKG_NAME=cfg-if CARGO_PKG_REPOSITORY='https://github.com/alexcrichton/cfg-if' CARGO_PKG_VERSION=0.1.10 CARGO_PKG_VERSION_MAJOR=0 CARGO_PKG_VERSION_MINOR=1 CARGO_PKG_VERSION_PATCH=10 CARGO_PKG_VERSION_PRE='' LD_LIBRARY_PATH='/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps:/home/glaubitz/tmp/usr/local/lib:' /home/glaubitz/tmp/usr/local/bin/rustc --crate-name cfg_if --edition=2018 /home/glaubitz/rustc/rustc-1.50.0+dfsg1/vendor/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=144fa4f9a0a129e5 -C extra-filename=-144fa4f9a0a129e5 --out-dir /home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps -L dependency=/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps --cap-lints warn -Cdebuginfo=2 -C linker=powerpc-linux-gnu-gcc -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings
{"artifact":"/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps/cfg_if-144fa4f9a0a129e5.d","emit":"dep-info"}
Illegal instruction
(sid_powerpc-dchroot)glaubitz@perotto:~/rustc/rustc-1.50.0+dfsg1$

[glaubitz]

I cannot get a sensible backtrace, unfortunately:

(gdb) r --crate-name cfg_if --edition=2018 /home/glaubitz/rustc/rustc-1.50.0+dfsg1/vendor/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=144fa4f9a0a129e5 -C extra-filename=-144fa4f9a0a129e5 --out-dir /home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps -L dependency=/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps --cap-lints warn -Cdebuginfo=2 -C linker=powerpc-linux-gnu-gcc -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings
Starting program: /home/glaubitz/tmp/usr/local/bin/rustc --crate-name cfg_if --edition=2018 /home/glaubitz/rustc/rustc-1.50.0+dfsg1/vendor/cfg-if-0.1.10/src/lib.rs --error-format=json --json=diagnostic-rendered-ansi,artifacts --crate-type lib --emit=dep-info,metadata,link -Cembed-bitcode=no -C debuginfo=2 -C metadata=144fa4f9a0a129e5 -C extra-filename=-144fa4f9a0a129e5 --out-dir /home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps -L dependency=/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps --cap-lints warn -Cdebuginfo=2 -C linker=powerpc-linux-gnu-gcc -Wrust_2018_idioms -Wunused_lifetimes -Dwarnings
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/powerpc-linux-gnu/libthread_db.so.1".
[New Thread 0xede3f0f0 (LWP 256221)]
{"artifact":"/home/glaubitz/rustc/rustc-1.50.0+dfsg1/build/bootstrap/debug/deps/cfg_if-144fa4f9a0a129e5.d","emit":"dep-info"}

Thread 2 "rustc" received signal SIGILL, Illegal instruction.
[Switching to Thread 0xede3f0f0 (LWP 256221)]
0xf4f59bbc in std::thread::local::fast::Key$LT$T$GT$::try_initialize::h23c87cba01958f05 ()
   from /home/glaubitz/tmp/usr/local/bin/../lib/librustc_driver-9cf50d6324ac9732.so
(gdb) bt
#0  0xf4f59bbc in std::thread::local::fast::Key$LT$T$GT$::try_initialize::h23c87cba01958f05 ()
   from /home/glaubitz/tmp/usr/local/bin/../lib/librustc_driver-9cf50d6324ac9732.so
Backtrace stopped: previous frame identical to this frame (corrupt stack?)
(gdb)

[glaubitz]

I'm bisecting this now. I think it might actually be a bug in the Rust compiler and not LLVM.

[glaubitz]

git bisect has lead me to 0328e69:

0328e69287b083af4d5d4b49cfc9175e9c82c88e is the first bad commit
commit 0328e69287b083af4d5d4b49cfc9175e9c82c88e
Author: Josh Triplett <josh@joshtriplett.org>
Date:   Sun Oct 25 22:11:20 2020 -0700

    Compile tools and internal libraries with the initial-exec TLS model
    
    This should produce more efficient code, with fewer calls to
    __tls_get_addr. The tradeoff is that libraries using it won't work with
    dlopen, but that shouldn't be a problem for tools or for our own
    internal libraries.
    
    Co-authored-by: Mark Rousskov <mark.simulacrum@gmail.com>

 src/bootstrap/builder.rs | 8 ++++++++
 src/bootstrap/lib.rs     | 4 ++++
 2 files changed, 12 insertions(+)

[nagisa]

At a first blush sounds like a limitation of the platform most likely (much like how it doesn't really support e.g. copy relocations all that well on PPC64) in which case a simple exclusion would make sense here.

[glaubitz]

I can confirm that reverting that change fixes the Rust compiler on 32-bit PowerPC for me. So, I agree, it should just be disabled on this target.

[glaubitz]

I can submit a simple patch either tonight or tomorrow some time.

[stefson]

@glaubitz there are reports of powerpc little endian toolchains still hitting this bug, can you imagine to push a simple patch to include the case of little endian as well?

example: https://github.com/void-ppc/void-packages/blob/master/srcpkgs/rust/patches/fix-ppc32.patch

[workingjubilee]

...Does LLVM, or any of our codegen backends for that matter, actually support PowerPC 32-bit LE targets? Do they really exist? I know PowerPCs are bi-endian, but I thought the endian-switch bit was an extension that 32-bit architectures didn't have to include.

Well, I do know we don't actually support any ppc32le Rust targets. I don't think it would exactly be what one would call a "maintenance burden" to accept a patch that adds a mild fix to bootstrap for custom targets. But if people are really using this target, it might be better to submit an application to add it as a new tier 3 target, so we can at least be Formally Aware of it.

[jrtc27]

Yes. https://reviews.llvm.org/D93918

[workingjubilee]

Fair enough!

@stefson The patch you ask for is simple so if you send it in we probably would happily accept it, but the best way to make sure Void Linux's powerpcle-unknown-linux-* targets are accounted for in the future would be to go through the process of upstreaming them as a tier 3 target. For Tier 3 Platform Support, it isn't required that it works well or passes tests. In fact, it isn't even required to build. However, merely having such support in the codebase would help with letting us know which tuples are actually in use, and thus reduce the likelihood of accidentally stomping on them.