Cannot build on RPi/aarch64; Rust returns E0685 error for unstable package 'stdsimd'

[mapachurro]

My goal: run filecoin on an 8GB Raspberry Pi 4. Space Race!
My environment: Raspbian OS 64-bit 'beta', Rust and Go most recent versions; running nightly Rust toolchain (more on this below), jq installed; lotus master branch; I've deleted and re-cloned the repo various times and the error re-occurs.
My issue: I get to the point of running 'make clean all' as per these instructions and I get an error code from Rust. I should say that at some point, I read some instructions that indicated I should be using additional flags to get it to compile properly on RPi:

export RUSTFLAGS="-C target-cpu=native -g"
export FFI_BUILD_FROM_SOURCE=1

So I have declared those flags, and when I run 'make clean all', I get a Rust error: E0685, "use of unstable library feature stdsimd". Rust helpfully directs me to this github issue (which is closed, and for whatever reason Github won't let me comment on the discussion of reopening it ), and gives me the option to ask for more information. When I do, it gives me this:

An unstable feature was used.

Erroneous code example:

#[repr(u128)] // error: use of unstable library feature 'repr128'
enum Foo {
    Bar(u64),
}

If you're using a stable or a beta version of rustc, you won't be able to use
any unstable features. In order to do so, please switch to a nightly version of
rustc (by using rustup).

If you're using a nightly version of rustc, just add the corresponding feature
to be able to use it:

#![feature(repr128)]

#[repr(u128)] // ok!
enum Foo {
    Bar(u64),
}

It was thanks to these instructions that I switched to a nightly toolchain install; I have been editing the cargo.toml file at ~/lotus/extern/filecoin-ffi/rust/Cargo.toml, trying to force Rust to use 'stdsimd' as per instructions in the Rust Unstable Book and the Cargo Manual's section on Features; I was hoping to be able to find a solution and come to you with a PR, but I'm not there yet. These are some of the errors that I get as a result:

(at first, I tried to 'cherry-pick' features as per the Cargo Manual, and I got this):

error: failed to parse manifest at `/home/pi/lotus/extern/filecoin-ffi/rust/Cargo.toml`

Caused by:
  invalid type: boolean `false`, expected a sequence for key `features.default-features`

(so then, I tried to just declare 'stdsimd' as a feature)

error: failed to parse manifest at `/home/pi/lotus/extern/filecoin-ffi/rust/Cargo.toml`

Caused by:
  Feature `features` includes `stdsimd` which is neither a dependency nor another feature

Which is pretty spectacular, as I'm literally following Rust's instructions as to how to enable this unstable feature. I'm sure I'm missing something, and I'll keep plugging away at it, but I wanted to raise the issue, as I know there's a fair amount of interest in deploying filecoin on RPi, and soon Raspberry Pi OS 64-bit will be an officially supported/released distro, so this could be a great tool in terms of increasing accessibility of the network.

If anyone could point me to a solution to this if it already exists, or could help me to understand how to modify the Rust code properly (I'm a total noob with Rust), I'd be happy to submit a PR if I get it running.

[ribasushi]

@mapachurro you need to apply a local patch to force "rust nightly": stable rust does not support intrinsics on non-x86

#2176 (comment)

cc @whyrusleeping

[mapachurro]

Hey @ribasushi thanks. I had actually found that thread previously and tried your patch--it hung at the patch if I remember correctly. But I will try again hopefully tonight.

[mapachurro]

OK! It was chugging along beautifully until it wasn't.

  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "blst/bindings" "-Wall" "-Wextra" "-std=gnu++11" "-o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/fil-blst/fil_blst.o" "-c" "fil-blst/fil_blst.cpp"
  cargo:warning=fil-blst/fil_blst.cpp:13:10: fatal error: xmmintrin.h: No such file or directory
  cargo:warning= #include <xmmintrin.h>
  cargo:warning=          ^~~~~~~~~~~~~
  cargo:warning=compilation terminated.
  exit code: 1

  --- stderr

  error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "blst/bindings" "-Wall" "-Wextra" "-std=gnu++11" "-o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/fil-blst/fil_blst.o" "-c" "fil-blst/fil_blst.cpp" with args "c++" did not execute successfully (status code exit code: 1).


warning: build failed, waiting for other jobs to finish...
error: build failed
+ rm -f /tmp/tmp.N0yMRr9xzJ
make[1]: *** [Makefile:11: .install-filcrypto] Error 101
make[1]: Leaving directory '/home/pi/lotus/extern/filecoin-ffi'
make: *** [Makefile:37: build/.filecoin-install] Error 2

And here's my uname output:

pi@raspberrypi:~/lotus $ uname -a
Linux raspberrypi 5.4.42-v8+ #1319 SMP PREEMPT Wed May 20 14:18:56 BST 2020 aarch64 GNU/Linux

So it is running aarch64.

Go:

pi@raspberrypi:~/lotus $ go version
go version go1.15.2 linux/arm64

Rust:

pi@raspberrypi:~/lotus $ rustc --version
rustc 1.48.0-nightly (fc2daaae6 2020-09-28)

Cheers.

[ribasushi]

@mapachurro this seems like you do not have https://packages.debian.org/unstable/libclang-dev + deps available. Please check and report.

[mapachurro]

It would seem I do not have it available:

pi@raspberrypi:~ $ sudo apt-get install libclang-9-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
You might want to run 'apt --fix-broken install' to correct these.
The following packages have unmet dependencies:
 libclang-9-dev : Depends: libobjc-8-dev but it is not going to be installed
                  Depends: libclang1-9 (= 1:9.0.1-6+rpi1~bpo10+1) but it is not going to be installed
                  Depends: libclang-common-9-dev (= 1:9.0.1-6+rpi1~bpo10+1) but it is not going to be installed
 libgl1-mesa-dev : Depends: libgl-dev
E: Unmet dependencies. Try 'apt --fix-broken install' with no packages (or specify a solution).
pi@raspberrypi:~ $ sudo apt --fix-broken install
Reading package lists... Done
Building dependency tree       
Reading state information... Done
Correcting dependencies... Done
The following additional packages will be installed:
  libgl-dev
The following NEW packages will be installed:
  libgl-dev
0 upgraded, 1 newly installed, 0 to remove and 145 not upgraded.
15 not fully installed or removed.
Need to get 0 B/100 kB of archives.
After this operation, 1,389 kB of additional disk space will be used.
Do you want to continue? [Y/n] Y
(Reading database ... 91728 files and directories currently installed.)
Preparing to unpack .../libgl-dev_1.3.0-7~bpo10+1_arm64.deb ...
Unpacking libgl-dev:arm64 (1.3.0-7~bpo10+1) ...
dpkg: error processing archive /var/cache/apt/archives/libgl-dev_1.3.0-7~bpo10+1_arm64.deb (--unpack):
 trying to overwrite '/usr/include/KHR/khrplatform.h', which is also in package libraspberrypi-dev 2+git20200520~143707+2fe4ca3-1
Errors were encountered while processing:
 /var/cache/apt/archives/libgl-dev_1.3.0-7~bpo10+1_arm64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

This is what happens when I specifically try to install libgl-dev:

pi@raspberrypi:~/lotus $ sudo apt-get install libgl-dev
Reading package lists... Done
Building dependency tree       
Reading state information... Done
The following NEW packages will be installed:
  libgl-dev
0 upgraded, 1 newly installed, 0 to remove and 145 not upgraded.
15 not fully installed or removed.
Need to get 0 B/100 kB of archives.
After this operation, 1,389 kB of additional disk space will be used.
(Reading database ... 91728 files and directories currently installed.)
Preparing to unpack .../libgl-dev_1.3.0-7~bpo10+1_arm64.deb ...
Unpacking libgl-dev:arm64 (1.3.0-7~bpo10+1) ...
dpkg: error processing archive /var/cache/apt/archives/libgl-dev_1.3.0-7~bpo10+1_arm64.deb (--unpack):
 trying to overwrite '/usr/include/KHR/khrplatform.h', which is also in package libraspberrypi-dev 2+git20200520~143707+2fe4ca3-1
Errors were encountered while processing:
 /var/cache/apt/archives/libgl-dev_1.3.0-7~bpo10+1_arm64.deb
E: Sub-process /usr/bin/dpkg returned an error code (1)

[ribasushi]

Uh-oh... this is a bug in raspi's package repo: having 2 packages sharing the same file is a big no-no.
I do not have bandwidth today, but would be very interested in a zoom session with screenshare to figure this all out some time Thursday. Would you be open to that?

[mapachurro]

Sounds awesome. I'll email you to coordinate.

[mapachurro]

I appear to have fixed the unmet dependencies issue by following these instructions.

I then went through the above error message that I had gotten, and installed all the dependencies for libclang-9-dev, up to and including libclang-9-dev. They all installed without a problem. However, I'm still getting the stderr. Here's the full output.

Hopefully we can hash some of this out over the weekend.

pi@raspberrypi:~/lotus $ RUSTFLAGS="-C target-cpu=native -g" FFI_BUILD_FROM_SOURCE=1 make all lotus-bench lotus-shed
make -C extern/filecoin-ffi/ .install-filcrypto
make[1]: Entering directory '/home/pi/lotus/extern/filecoin-ffi'
./install-filcrypto
+ auth_header=()
+ '[' -n '' ']'
++ dirname ./install-filcrypto
+ cd .
+ rust_sources_dir=rust
++ jq -r '.[].check_cpu_for_feature | select(. != null)'
+ cpu_features_required_for_optimized_release='adx
sha_ni
sha2
sse2
avx2
avx
sse4_2
sse4_1'
+ main
++ get_release_flags
++ local __features=
++ [[ ! -f /proc/cpuinfo ]]
+++ uname -a
+++ grep x86_64
++ x86_64_uname=
++ '[' -n '' ']'
+++ cat /proc/cpuinfo
+++ grep Features
+++ head -n 1
++ __features='Features	: fp asimd evtstrm crc32 cpuid'
++ feature_map=("adx:+adx" "sha_ni:+sha" "sha2:+sha2" "sse2:+sse2" "avx2:+avx2" "avx:+avx" "sse4_2:+sse4.2" "sse4_1:+sse4.1")
++ target_features=
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c adx
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c sha_ni
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c sha2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c sse2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ grep -c avx2
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c avx
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c sse4_2
++ current_feature=0
++ '[' 1 = 0 ']'
++ for x in ${cpu_features_required_for_optimized_release[@]}
+++ echo 'Features	: fp asimd evtstrm crc32 cpuid'
+++ grep -c sse4_1
++ current_feature=0
++ '[' 1 = 0 ']'
++ echo ''
+ local __release_flags=
+ '[' 1 '!=' 1 ']'
+ echo '[install-filcrypto/main] building libfilcrypto from local sources (dir = rust)'
[install-filcrypto/main] building libfilcrypto from local sources (dir = rust)
+ build_from_source filcrypto rust ''
+ local __library_name=filcrypto
+ local __rust_sources_path=rust
+ local __release_flags=
++ git rev-parse HEAD
+ local __repo_sha1=f640612a1a1f7a2dd8b3a49e1531db0aa0f63447
+ local __repo_sha1_truncated=f640612a1a1f7a2d
+ echo 'building from source @ f640612a1a1f7a2d'
building from source @ f640612a1a1f7a2d
++ command -v cargo
+ '[' -x /home/pi/.cargo/bin/cargo ']'
++ command -v rustup
+ '[' -x /home/pi/.cargo/bin/rustup ']'
+ pushd rust
~/lotus/extern/filecoin-ffi/rust ~/lotus/extern/filecoin-ffi
+ cargo --version
cargo 1.48.0-nightly (05c611ae3 2020-09-23)
+ '[' -n '' ']'
++ cat rust-toolchain
+ ./scripts/build-release.sh filcrypto nightly
+ main filcrypto nightly
+ [[ -z filcrypto ]]
+ [[ -z nightly ]]
++ mktemp
+ __build_output_log_tmp=/tmp/tmp.hrYwvCDkk8
+ trap '{ rm -f $__build_output_log_tmp; }' EXIT
+ local '__rust_flags=--print native-static-libs -C target-cpu=native -g'
+ RUSTFLAGS='--print native-static-libs -C target-cpu=native -g'
+ cargo +nightly build --release
+ tee /tmp/tmp.hrYwvCDkk8
   Compiling fil-blst v0.1.1
   Compiling filcrypto v0.7.5 (/home/pi/lotus/extern/filecoin-ffi/rust)
   Compiling reqwest v0.10.8
   Compiling phase21 v0.8.0
The following warnings were emitted during compilation:

warning: fil-blst/fil_blst.cpp:13:10: fatal error: xmmintrin.h: No such file or directory
warning:  #include <xmmintrin.h>
warning:           ^~~~~~~~~~~~~
warning: compilation terminated.

error: failed to run custom build command for `fil-blst v0.1.1`

Caused by:
  process didn't exit successfully: `/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-74b894e8c809170c/build-script-build` (exit code: 1)
  --- stdout
  Using fil-blst source directory "fil-blst"
  Using       blst source directory "blst"
  TARGET = Some("aarch64-unknown-linux-gnu")
  HOST = Some("aarch64-unknown-linux-gnu")
  CC_aarch64-unknown-linux-gnu = None
  CC_aarch64_unknown_linux_gnu = None
  HOST_CC = None
  CC = None
  CFLAGS_aarch64-unknown-linux-gnu = None
  CFLAGS_aarch64_unknown_linux_gnu = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  CC_aarch64-unknown-linux-gnu = None
  CC_aarch64_unknown_linux_gnu = None
  HOST_CC = None
  CC = None
  CFLAGS_aarch64-unknown-linux-gnu = None
  CFLAGS_aarch64_unknown_linux_gnu = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  CC_aarch64-unknown-linux-gnu = None
  CC_aarch64_unknown_linux_gnu = None
  HOST_CC = None
  CC = None
  CFLAGS_aarch64-unknown-linux-gnu = None
  CFLAGS_aarch64_unknown_linux_gnu = None
  HOST_CFLAGS = None
  CFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-Wall" "-Wextra" "-Wno-unused-command-line-argument" "-o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/blst/src/server.o" "-c" "blst/src/server.c"
  exit code: 0
  running: "cc" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-Wall" "-Wextra" "-Wno-unused-command-line-argument" "-o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/blst/build/assembly.o" "-c" "blst/build/assembly.S"
  exit code: 0
  AR_aarch64-unknown-linux-gnu = None
  AR_aarch64_unknown_linux_gnu = None
  HOST_AR = None
  AR = None
  running: "ar" "crs" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/libblst.a" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/blst/src/server.o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/blst/build/assembly.o"
  exit code: 0
  cargo:rustc-link-lib=static=blst
  cargo:rustc-link-search=native=/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out
  TARGET = Some("aarch64-unknown-linux-gnu")
  HOST = Some("aarch64-unknown-linux-gnu")
  CXX_aarch64-unknown-linux-gnu = None
  CXX_aarch64_unknown_linux_gnu = None
  HOST_CXX = None
  CXX = None
  CXXFLAGS_aarch64-unknown-linux-gnu = None
  CXXFLAGS_aarch64_unknown_linux_gnu = None
  HOST_CXXFLAGS = None
  CXXFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  DEBUG = Some("false")
  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  CXX_aarch64-unknown-linux-gnu = None
  CXX_aarch64_unknown_linux_gnu = None
  HOST_CXX = None
  CXX = None
  CXXFLAGS_aarch64-unknown-linux-gnu = None
  CXXFLAGS_aarch64_unknown_linux_gnu = None
  HOST_CXXFLAGS = None
  CXXFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  CXX_aarch64-unknown-linux-gnu = None
  CXX_aarch64_unknown_linux_gnu = None
  HOST_CXX = None
  CXX = None
  CXXFLAGS_aarch64-unknown-linux-gnu = None
  CXXFLAGS_aarch64_unknown_linux_gnu = None
  HOST_CXXFLAGS = None
  CXXFLAGS = None
  CRATE_CC_NO_DEFAULTS = None
  CARGO_CFG_TARGET_FEATURE = Some("crc,crypto,fp,neon")
  running: "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "blst/bindings" "-Wall" "-Wextra" "-std=gnu++11" "-o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/fil-blst/fil_blst.o" "-c" "fil-blst/fil_blst.cpp"
  cargo:warning=fil-blst/fil_blst.cpp:13:10: fatal error: xmmintrin.h: No such file or directory
  cargo:warning= #include <xmmintrin.h>
  cargo:warning=          ^~~~~~~~~~~~~
  cargo:warning=compilation terminated.
  exit code: 1

  --- stderr


  error occurred: Command "c++" "-O3" "-ffunction-sections" "-fdata-sections" "-fPIC" "-I" "blst/bindings" "-Wall" "-Wextra" "-std=gnu++11" "-o" "/home/pi/lotus/extern/filecoin-ffi/rust/target/release/build/fil-blst-9d89b41304adb8fb/out/fil-blst/fil_blst.o" "-c" "fil-blst/fil_blst.cpp" with args "c++" did not execute successfully (status code exit code: 1).


warning: build failed, waiting for other jobs to finish...
^C++ rm -f /tmp/tmp.hrYwvCDkk8
make[1]: *** [Makefile:11: .install-filcrypto] Interrupt
make: *** [Makefile:37: build/.filecoin-install] Interrupt

[ribasushi]

@mapachurro sorry about dropping this for a while: there was so much churn in the underlying rust libs, that trying to tackle this wasn't feasible. However now the time seems right, and we will try to get the core proofs team to look at this in the coming days. Please track filecoin-project/rust-fil-proofs#1351 as a start, it is a hard-blocker to have your issue resolved.

[mapachurro]

Updating this issue:

I read the good news on filecoin-project/rust-fil-proofs#1351, which I understood to mean that if I start over with a fresh clone of the lotus repo, it should compile.

Some things looked better than they did before (libclang deps, for example), but I got this in the end:

# github.com/filecoin-project/filecoin-ffi/generated
/usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto
/usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto
/usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto
/usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto
/usr/bin/ld: skipping incompatible extern/filecoin-ffi/generated/../libfilcrypto.a when searching for -lfilcrypto
/usr/bin/ld: cannot find -lfilcrypto
collect2: error: ld returned 1 exit status
make: *** [Makefile:68: lotus] Error 2

Steps to reproduce:

-Following current instructions on https://docs.filecoin.io/get-started/lotus/installation/#software-dependencies, install Ubuntu/Debian dependencies, Rust, and Go
-rm -r current lotus directory
-clone lotus repo
-cd in, make sure I'm on master branch
-make clean all

My next thought is to try the target-cpu-native and build-from-source flags, if that fixes I will report back.

Thanks for all the awesome work so far!

[mapachurro]

Ok, with target-cpu-native and build-from-source, I get the following error:

   Compiling reqwest v0.10.8
error[E0308]: mismatched types
   --> /home/pi/.cargo/registry/src/git.luolix.top-1ecc6299db9ec823/bellperson-0.11.0/src/groth16/multiscalar.rs:268:19
    |
265 | fn prefetch<T>(p: *const T) {
    |             - this type parameter
...
268 |         _prefetch(p, _PREFETCH_READ, _PREFETCH_LOCALITY3);
    |                   ^ expected `i8`, found type parameter `T`
    |
    = note: expected raw pointer `*const i8`
               found raw pointer `*const T`

error: aborting due to previous error

For more information about this error, try `rustc --explain E0308`.
error: could not compile `bellperson`

To learn more, run the command again with --verbose.
warning: build failed, waiting for other jobs to finish...
error: build failed
+ rm -f /tmp/tmp.FoTnrFotuw
make[1]: *** [Makefile:11: .install-filcrypto] Error 101
make[1]: Leaving directory '/home/pi/lotus/extern/filecoin-ffi'
make: *** [Makefile:37: build/.filecoin-install] Error 2

[ribasushi]

@mapachurro the changes you linked have not been merged into lotus/master yet. You can try with release/v1.2.0, but it is not quite ready just yet.

[ribasushi]

@mapachurro at last! lotus v1.2.1 now builds and works on aarch64 out of the box ( no patches needed )

[mapachurro]

Oh that's very exciting! I will try tomorrow. Thanks to you and those doing the heavy lifting!!

[ribasushi]

@mapachurro if you have cycles to spare: the dev team would really love a contribution to the CI to build and run the tests under aarch64. We do not have sufficient expertise to deal with the qemu dance, but also do not like breaking this over and over :(
A CI entry would be very welcome!

cc @dignifiedquire

[mapachurro]

I'd love to, and I hear what you're saying about having a physical box rather than a virtual. Do you have some documentation as to what specifically you're looking for?

[ribasushi]

@mapachurro oh, I wasn't implying "physical box". Rather: there got to be some sort of black magic to make such an addition to circle-ci, just nobody has done the necessary research. So if you have the cycles... :)

[ribasushi]

@mapachurro I switched some of my personal builds to aarch64, with no issues. Closing this issue for now, please reopen if you still encounter problems.
Also please look into trying your hand at CI: we really need this :(