AArch64 bare-metal targets: Build rust-std

[andre-richter]

This PR complements #68253

[rust-highfive]

r? @pietroalbini

(rust_highfive has picked a reviewer for you, use r? to override)

[japaric]

Looks good to me. Thanks!

@bors r+

[bors]

📌 Commit 50f3b5d has been approved by japaric

[pietroalbini]

r? @japaric

[JohnTitor]

Seems failed in #68384 (comment)
@bors r-

[andre-richter]

Are the CC_aarch64_unknown_none and CC_aarch64_unknown_none_softloat in the Dockerfile wrong?

[andre-richter]

I removed the CC_ envs. Kindly review again.

[japaric]

Ah, sorry. I overlooked the contents of the CC variables. If the cc crate does not automatically pick a suitable gcc compiler for these targets (because it doesn't know about them) then you need to set e.g. CC_aarch64_unknown_none to a valid gcc toolchain like arm-none-eabi-gcc. However, I'm not sure the arm-none-eabi-gcc toolchain can generate machine code for AArch64 so you may need to install a different toolchain in the Docker image (arm64-something-gcc, maybe?).

Also, if the default settings of the gcc toolchain do not match the codegen options of rustc (e.g. arm-none-eabi-gcc generates ARMv4 by default but rustc codegen produces Thumb instructions that run on ARMv7-M) then you'll need to set CFLAGS_$TARGET_NAME to something like -march=armv8. Check what the cc crate does for aarch64-unknown-linux-gnu and use that as a reference to pick the values of CC_ and CFLAGS_.

You can test the part of the build that's failing, without bootstrapping the compiler, by running xargo build --target aarch64-unknown-none --features c -vv on the compiler-builtins crate that's on crates.io (you may also need to set the COMPILER_RT_ROOT env var to the path to the C source code of the compiler-rt project -- this is included in the rust-lang/rust repo). If you set the CC and CFLAGS vars correctly that xargo command should complete successfully.

[pietroalbini]

Also, you can test the builder locally with:

src/ci/docker/run.sh dist-various-1

[andre-richter]

Thanks guys, that helped a lot to understand what is going on in the Docker container. Appreciate it.

So I tried with Linaro's aarch64 bare-metal gcc:

https://releases.linaro.org/archive/15.06/components/toolchain/binaries/4.8/aarch64-elf/

Compiling compiler-builtins using xargo and with RUST_COMPILER_RT_ROOT like @japaric mentioned works for aarch64-unknown-none. This is the target with hardware FP enabled.

However, problems arise with aarch64-unknown-none-softfloat, where we disable FP codegen.

Kicking off an equivalent build of compiler-builtins fails because it tries to compile floating point code:

CFLAGS_aarch64_unknown_none_softfloat="-mstrict-align -march=armv8-a+nofp+nosimd" CC_aarch64_unknown_none_softfloat=$PWD/gcc/bin/aarch64-elf-gcc RUST_COMPILER_RT_ROOT=/opt/repos/rust/src/llvm-project/compiler-rt xargo build --target aarch64-unknown-none-softfloat --features c -vv

// Some omitted

[compiler_builtins 0.1.24] running: "/opt/compiler-builtins/gcc/bin/aarch64-elf-gcc" "/opt/compiler-builtins/gcc/bin/aarch64-elf-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-mstrict-align" "-march=armv8-a+nofp+nosimd" "-fno-builtin" "-fvisibility=hidden" "-ffreestanding" "-DVISIBILITY_HIDDEN" "-o" "/opt/compiler-builtins/target/aarch64-unknown-none-softfloat/debug/build/compiler_builtins-2375755c04f9af94/out/comparetf2.o" "-c" "/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c"
[compiler_builtins 0.1.24] cargo:warning=In file included from /opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:40:0:
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/fp_lib.h: In function 'toRep':
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/fp_lib.h:218:23: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning= static __inline rep_t toRep(fp_t x) {
[compiler_builtins 0.1.24] cargo:warning=                       ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/fp_lib.h:222:5: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   } rep = {.f = x};
[compiler_builtins 0.1.24] cargo:warning=     ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c: In function '__letf2':
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:45:32: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning= COMPILER_RT_ABI enum LE_RESULT __letf2(fp_t a, fp_t b) {
[compiler_builtins 0.1.24] cargo:warning=                                ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:45:32: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:47:28: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   const srep_t aInt = toRep(a);
[compiler_builtins 0.1.24] cargo:warning=                            ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:48:28: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   const srep_t bInt = toRep(b);
[compiler_builtins 0.1.24] cargo:warning=                            ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c: In function '__getf2':
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:98:32: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning= COMPILER_RT_ABI enum GE_RESULT __getf2(fp_t a, fp_t b) {
[compiler_builtins 0.1.24] cargo:warning=                                ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:98:32: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:100:28: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   const srep_t aInt = toRep(a);
[compiler_builtins 0.1.24] cargo:warning=                            ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:101:28: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   const srep_t bInt = toRep(b);
[compiler_builtins 0.1.24] cargo:warning=                            ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c: In function '__unordtf2':
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:128:21: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning= COMPILER_RT_ABI int __unordtf2(fp_t a, fp_t b) {
[compiler_builtins 0.1.24] cargo:warning=                     ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:128:21: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:129:27: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   const rep_t aAbs = toRep(a) & absMask;
[compiler_builtins 0.1.24] cargo:warning=                           ^
[compiler_builtins 0.1.24] cargo:warning=/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c:130:27: sorry, unimplemented: '-mgeneral-regs-only' and floating point code
[compiler_builtins 0.1.24] cargo:warning=   const rep_t bAbs = toRep(b) & absMask;
[compiler_builtins 0.1.24] cargo:warning=                           ^
[compiler_builtins 0.1.24] exit code: 1
[compiler_builtins 0.1.24] 
[compiler_builtins 0.1.24] 
[compiler_builtins 0.1.24] error occurred: Command "/opt/compiler-builtins/gcc/bin/aarch64-elf-gcc" "/opt/compiler-builtins/gcc/bin/aarch64-elf-gcc" "-O0" "-ffunction-sections" "-fdata-sections" "-fPIC" "-g" "-fno-omit-frame-pointer" "-mstrict-align" "-march=armv8-a+nofp+nosimd" "-fno-builtin" "-fvisibility=hidden" "-ffreestanding" "-DVISIBILITY_HIDDEN" "-o" "/opt/compiler-builtins/target/aarch64-unknown-none-softfloat/debug/build/compiler_builtins-2375755c04f9af94/out/comparetf2.o" "-c" "/opt/rust/src/llvm-project/compiler-rt/lib/builtins/comparetf2.c" with args "aarch64-elf-gcc" did not execute successfully (status code exit code: 1).
[compiler_builtins 0.1.24] 
[compiler_builtins 0.1.24] 
error: failed to run custom build command for `compiler_builtins v0.1.24 (/opt/compiler-builtins)`

Any knobs we can turn here to prevent the FP code compilation?
Probably a patch to build.rs in compiler-builtins is needed?

[japaric]

Kicking off an equivalent build of compiler-builtins fails because it tries to compile floating point code:

Hmm, I don't see any errors there; just a warning. If C code's not compiling then you port it to Rust to fix the problem. If assembly code's failing to compile it could be that it should not be compiled in the first place because the target never uses the intrinsic; in that case you should modify compiler-builtins' build.rs to not build that assembly file / intrinsic for the soft float target. Or it could be the case that the assembly has some #ifdef pre-processor thing and the build.rs needs to define (see cc API docs) something to make it compile for the soft float target.

In any case, that needs to be fixed in compiler-builtins. Which means the soft float is blocked on a new crates.io release of that crate. On the other hand, the other aarch64 target can land now. IOW, you can modify this PR to land one target now and leave the other target for a follow up PR.

[andre-richter]

Unfortunately, I am lacking background on both compiler intrinsics knowledge and expectations with respect to what compilers like GCC are supposed to do with C code that "uses" floating point when the flags -march=armv8-a+nofp+nosimd are set. I need a little more guidance.

In the ARM Compiler 6.13 reference, I found this note:

There are no software floating-point libraries for targets in AArch64 state. At link time armlink links against AArch64 library code that can use floating-point and SIMD instructions and registers. This still applies if you compile the source with -march=+nofp+nosimd to prevent the compiler from using floating-point and SIMD instructions and registers.

To prevent the use of any floating-point instruction or register, either re-implement the library functions or create your own library that does not use floating-point instructions or registers.

This does not seem to be exactly what I observe with the Linaro bare-metal aarch64 GCC as shown in #68334 (comment). GCC bails out as soon as it hits FP code. For example, on definition of a long double or when an instruction for using a float register is about to be emitted.

My question is, what is the idiomatic "ARM-way" of solving this? Just not compiling any compiler-rt files emitting FP code? Soft-float library (doesn't seem to exist, at least for ARM's compiler)?
Also not sure if I got the "port to Rust if compilation fails". Because it would pull in its own soft-float library?

CC @parched, would appreciate an additional opinion.

[raw-bin]

Hi Andre.

(Thanks for reaching out by the side!)

This one does come up occasionally.

The standard 64-bit Arm ARM - (the architecture reference manual) could give brave readers (it's a massive doc!) the impression that a processor implementation without hardware floating point support is permissible.

In reality, however, the Arm 64-bit PCS (Procedure Call Standard) mandates the use of a hardware floating point unit.

This is a universal expectation and any deviations are basically non-standard~broken.

In rare instances we have had partners make cases for going the non-hardware-fp way (for time-to-market or power-perf-area reasons) but no one implements such processors because of the cost of maintaining the corresponding non-standard toolchains and software.

Looking at the thread - and assuming I haven't misunderstood things - I would say that the idiomatic thing to do would be to not have a aarch64-unknown-none-softfloat compiler-builtins target.

Let me know if any further clarification is needed. Happy to try and help!

[andre-richter]

Had a chat with @raw-bin offline. We are still struggling to understand if it can make sense to exclude all the compiler-rt files that would make an aarch64-gcc-elf with +nofp fail in order to arrive at a sane build of the library.

@japaric, the following files are the offending ones I identified. You know this topic a lot better. Please tell us what you think.

diff --git a/build.rs b/build.rs
index b520b62..56fe410 100644
--- a/build.rs
+++ b/build.rs
@@ -51,7 +51,7 @@ fn main() {
         //   time). This can probably be removed in the future
         if !target.contains("wasm32") && !target.contains("nvptx") && !target.starts_with("riscv") {
             #[cfg(feature = "c")]
-            c::compile(&llvm_target);
+            c::compile(&llvm_target, &target);
         }
     }
 
@@ -121,13 +121,24 @@ mod c {
     }
 
     /// Compile intrinsics from the compiler-rt C source code
-    pub fn compile(llvm_target: &[&str]) {
+    pub fn compile(llvm_target: &[&str], target: &String) {
         let target_arch = env::var("CARGO_CFG_TARGET_ARCH").unwrap();
         let target_env = env::var("CARGO_CFG_TARGET_ENV").unwrap();
         let target_os = env::var("CARGO_CFG_TARGET_OS").unwrap();
         let target_vendor = env::var("CARGO_CFG_TARGET_VENDOR").unwrap();
+        let mut use_float = true;
         let cfg = &mut cc::Build::new();
 
+        // Evaluate if files containing floating point code should be considered.
+        if target_arch == "aarch64" {
+            let cflags_key = String::from("CFLAGS_") + &(target.to_owned().replace("-", "_"));
+            if let Ok(cflags_value) = env::var(cflags_key) {
+                if cflags_value.contains("+nofp") || cflags_value.contains("+nosimd") {
+                    use_float = false;
+                }
+            }
+        }
+
         cfg.warnings(false);
 
         if target_env == "msvc" {
@@ -166,34 +177,39 @@ mod c {
             ("__cmpdi2", "cmpdi2.c"),
             ("__ctzdi2", "ctzdi2.c"),
             ("__ctzsi2", "ctzsi2.c"),
-            ("__divdc3", "divdc3.c"),
-            ("__divsc3", "divsc3.c"),
-            ("__divxc3", "divxc3.c"),
-            ("__extendhfsf2", "extendhfsf2.c"),
             ("__int_util", "int_util.c"),
-            ("__muldc3", "muldc3.c"),
-            ("__mulsc3", "mulsc3.c"),
             ("__mulvdi3", "mulvdi3.c"),
             ("__mulvsi3", "mulvsi3.c"),
-            ("__mulxc3", "mulxc3.c"),
-            ("__negdf2", "negdf2.c"),
             ("__negdi2", "negdi2.c"),
-            ("__negsf2", "negsf2.c"),
             ("__negvdi2", "negvdi2.c"),
             ("__negvsi2", "negvsi2.c"),
             ("__paritydi2", "paritydi2.c"),
             ("__paritysi2", "paritysi2.c"),
             ("__popcountdi2", "popcountdi2.c"),
             ("__popcountsi2", "popcountsi2.c"),
-            ("__powixf2", "powixf2.c"),
             ("__subvdi3", "subvdi3.c"),
             ("__subvsi3", "subvsi3.c"),
-            ("__truncdfhf2", "truncdfhf2.c"),
-            ("__truncdfsf2", "truncdfsf2.c"),
-            ("__truncsfhf2", "truncsfhf2.c"),
             ("__ucmpdi2", "ucmpdi2.c"),
         ]);
 
+        if use_float {
+            sources.extend(&[
+                ("__divdc3", "divdc3.c"),
+                ("__divsc3", "divsc3.c"),
+                ("__divxc3", "divxc3.c"),
+                ("__extendhfsf2", "extendhfsf2.c"),
+                ("__muldc3", "muldc3.c"),
+                ("__mulsc3", "mulsc3.c"),
+                ("__mulxc3", "mulxc3.c"),
+                ("__negdf2", "negdf2.c"),
+                ("__negsf2", "negsf2.c"),
+                ("__powixf2", "powixf2.c"),
+                ("__truncdfhf2", "truncdfhf2.c"),
+                ("__truncdfsf2", "truncdfsf2.c"),
+                ("__truncsfhf2", "truncsfhf2.c"),
+            ]);
+        }
+
         // When compiling in rustbuild (the rust-lang/rust repo) this library
         // also needs to satisfy intrinsics that jemalloc or C in general may
         // need, so include a few more that aren't typically needed by
@@ -214,12 +230,15 @@ mod c {
                 ("__ffsti2", "ffsti2.c"),
                 ("__mulvti3", "mulvti3.c"),
                 ("__negti2", "negti2.c"),
-                ("__negvti2", "negvti2.c"),
                 ("__parityti2", "parityti2.c"),
                 ("__popcountti2", "popcountti2.c"),
                 ("__subvti3", "subvti3.c"),
                 ("__ucmpti2", "ucmpti2.c"),
             ]);
+
+            if use_float {
+                sources.extend(&[("__negvti2", "negvti2.c")]);
+            }
         }
 
         if target_vendor == "apple" {
@@ -372,7 +391,7 @@ mod c {
             ]);
         }
 
-        if target_arch == "aarch64" {
+        if target_arch == "aarch64" && use_float {
             sources.extend(&[
                 ("__comparetf2", "comparetf2.c"),
                 ("__extenddftf2", "extenddftf2.c"),

[andre-richter]

I had looked at this part of the code briefly, but couldn’t see why it would early return for the -softfloat target but continue for the other one 🤔

[alexcrichton]

Oh that I believe is due to this conditional which checks for the string -none-, not -none

[andre-richter]

Aaaaah thanks for this pointer. Wouldn't have found that anytime soon.

Let me push a fix for that. After all, we want both targets to match for no_std.

[andre-richter]

All PRs are merged now that were needed.

A local run of src/ci/docker/run.sh dist-various-1 I just did was successful. I think we can give this another try now.

[andre-richter]

Ping ☺️

[Dylan-DPC-zz]

@japaric this is ready to review

[andre-richter]

@rustbot modify labels: WG-embedded

[japaric]

@andre-richter sorry for missing the pings! Thanks again for pushing this to the finish line.

@bors r+

[bors]

📌 Commit 176bf3c has been approved by japaric

[bors]

☔ The latest upstream changes (presumably #70726) made this pull request unmergeable. Please resolve the merge conflicts.