Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Tracking Issue for Aarch64 Images #975

Open
Alexhuszagh opened this issue Jul 27, 2022 · 25 comments
Open

Tracking Issue for Aarch64 Images #975

Alexhuszagh opened this issue Jul 27, 2022 · 25 comments
Labels
A-aarch64-host Area: ARMv8 hosts container-images tracking-issue Issue to track progress for a certain feature/enhancement..

Comments

@Alexhuszagh
Copy link
Contributor

Alexhuszagh commented Jul 27, 2022

This is a tracking issue for building aarch64 images, meaning they run natively on linux/arm64 instead of linux/amd64, and what's confirmed to work, what needs to be fixed, and what's currently untested, so we can ensure the maintainers don't duplicate work, new contributors know where to start, and looking at what to patch for future builds. We can also likely add builds for PPC64LE, since almost all of the maintained packages that have install candidates for ARM64 also support PPC64LE. Currently, most of the linux GNU packages are/will be failing.

Note that the PPC64LE images are currently untested, but assumed to work.

Currently Succeeding

Builds of these images for aarch64 can be found here for debugging and testing purposes (these are not official images, not are they guaranteed to be stable).

  • aarch64-unknown-linux-gnu (x86_64, i686, arm64, ppc64le, depends on Add concept of a native Dockerfile. #982)
  • s390x-unknown-linux-gnu (x86_64, i686, arm64, pp64le)
  • arm-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
  • armv5te-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
  • armv7-unknown-linux-gnueabi (x86_64, i686, arm64, pp64le)
  • armv7-unknown-linux-gnueabihf (x86_64, i686, arm64, pp64le)
  • i586-unknown-linux-gnu (x86_64, arm64, pp64le)
  • i686-unknown-linux-gnu (x86_64, i686, arm64, pp64le)
  • powerpc64le-unknown-linux-gnu (x86_64, i686, arm64)
  • riscv64gc-unknown-linux-gnu (x86_64, i686, arm64, ppc64le)
  • x86_64-unknown-linux-gnu (x86_64, i686, arm64, ppc64le, depends on Add concept of a native Dockerfile. #982)
  • aarch64-unknown-linux-musl (likely all architectures)
  • arm-unknown-linux-musleabi (likely all architectures)
  • arm-unknown-linux-musleabihf (likely all architectures)
  • armv5te-unknown-linux-musleabi (likely all architectures)
  • armv7-unknown-linux-musleabi (likely all architectures)
  • armv7-unknown-linux-musleabihf (likely all architectures)
  • i586-unknown-linux-musl (likely all architectures)
  • i686-unknown-freebsd (likely all architectures)
  • mips-unknown-linux-musl (likely all architectures)
  • mips64-unknown-linux-muslabi64 (likely all architectures)
  • mips64el-unknown-linux-muslabi64 (likely all architectures)
  • mipsel-unknown-linux-musl (likely all architectures)
  • sparcv9-sun-solaris (likely all architectures)
  • thumbv6m-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
  • thumbv7em-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
  • thumbv7em-none-eabihf (x86_64, armhf, arm64, pp64le, s390x)
  • thumbv7m-none-eabi (x86_64, armhf, arm64, pp64le, s390x)
  • thumbv7neon-unknown-linux-gnueabihf (x86_64, armhf, arm64, pp64le, s390x)
  • x86_64-unknown-freebsd (likely all architectures)

Currently Failing

  • arm-unknown-linux-gnueabihf (Unknown error in crosstool-ng build process)
  • sparc64-unknown-linux-gnu (Unable to locate package g++-sparc64-linux-gnu, not available for ARM64, will need a full compile from source)
  • mips-unknown-linux-gnu (Package for ARM64 and PPC64LE only in 22.04+, not in 20.04)
  • mips64-unknown-linux-gnuabi64 (Unable to locate package g++-mips64-linux-gnuabi64, not available for ARM64, will need a full compile from source)
  • mips64el-unknown-linux-gnuabi64 (Unable to locate package g++-mips64el-linux-gnuabi64, not available for ARM64, will need a full compile from source)
  • mipsel-unknown-linux-gnu (Unable to locate package g++-mipsel-linux-gnu, not available for ARM64, will need a full compile from source)
  • powerpc-unknown-linux-gnu (Unable to locate package g++-powerpc-linux-gnu, not available for ARM64, only x86_64 and PPC64LE, will need a full compile from source)
  • powerpc64-unknown-linux-gnu (Unable to locate package g++-powerpc64-linux-gnu, not available for ARM64, will need a full compile from source)
  • i686-pc-windows-gnu (unlikely to work without building WINE from source)
  • x86_64-pc-windows-gnu (unlikely to work without building WINE from source)
  • wasm32-unknown-emscripten (no linux/arm64 image for emsdk)
  • x86_64-sun-solaris (fails while building GCC).

Failing, Cannot Fix

  • aarch64-linux-android
  • arm-linux-androideabi
  • armv7-linux-androideabi
  • i686-linux-android
  • thumbv7neon-linux-androideabi
  • x86_64-linux-android

The Android images have the following requirements, which means it will be impossible to build a cross-compiler for them without significant effort from a non-x86 host:

  1. They require gcc-multilib and g++-multilib, which are not available on aarch64 and any host other than i686, x86_64, and s390x.
  2. They use prebuilts for clang, gcc and go that assume an x86 host.
  3. The NDKs assume an x86_64 host.

The only solution therefore would be to rebuild the entire NDK from source, get Android system, replace the prebuilts with native toolchains for a non-x86 host, update code so it uses the correct non-x86 prebuilts, etc. If this sounds like an issue, it is.

Currently Untested

  • x86_64-unknown-dragonfly
  • x86_64-unknown-illumos
  • x86_64-unknown-linux-gnu:centos
  • x86_64-unknown-linux-musl
  • x86_64-unknown-netbsd

Contributing

If you'd like to contribute your own tests, successes, etc., feel free to do so with the following examples. The runners, std, C++, and dynamic library support can be found in the Github actions ci.yml.

# change this for specific targets
$ export TARGET=s390x-unknown-linux-gnu
$ cargo build-docker-image --platform linux/arm64/v8=aarch64-unknown-linux-gnu $TARGET
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu"
$ export CROSS_TARGET_S390X_UNKNOWN_LINUX_GNU_IMAGE="ghcr.io/cross-rs/$TARGET:local"
$ export STD=1
$ export CPP=1
$ export DYLIB=1
$ export RUN=1
$ export RUNNERS="qemu-user qemu-system"
$ ci/test.sh
@Alexhuszagh
Copy link
Contributor Author

If we install packages from Debian for some of these toolchains, we can support a lot more architectures. Potential solutions include:

  • mips64-unknown-linux-gnuabi64: (amd64 arm64 i386 mips64el mipsel ppc64el)
  • mips64el-unknown-linux-gnuabi64: (amd64 arm64 i386 mipsel ppc64el)
  • mipsel-unknown-linux-gnu: (amd64 arm64 i386 mips64el ppc64el)

This only seems to affect MIPS however.

@DoumanAsh
Copy link

DoumanAsh commented Jul 29, 2022

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

@Alexhuszagh
Copy link
Contributor Author

Alexhuszagh commented Jul 29, 2022

Just observation I encountered issue with latest libc crate (0.2.126) requiring newer glib version

  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.33' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.32' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)
  /target/release/build/libc-542d813a522391b3/build-script-build: /lib/x86_64-linux-gnu/libc.so.6: version `GLIBC_2.34' not found (required by /target/release/build/libc-542d813a522391b3/build-script-build)

It would be good to consider bumping ubuntu images as they have relatively old glib for cross compilation

That's unrelated (and an upstream bug): see #724. Try running cargo clean and then rebuilding. This can occur in any repository with a build script. It can happen fairly regularly if switching between toolchains. This is a cargo bug, but is exacerbated by how cross functions (swapping toolchains with the same target directory, which causes the bug to manifest).

@Emilgardis

This comment was marked as off-topic.

@abcd-ca
Copy link

abcd-ca commented Nov 5, 2022

I downloaded an aarch64 image to my host (my M1 mac). Would someone tell me how to use it? I've got a knowledge gap between having the native (M1, aarch64) image and this command:
CROSS_REMOTE=1 DOCKER_HOST=tcp://docker:2375/ cross build --release --target=armv7-unknown-linux-musleabihf.

I think for me the DOCKER_HOST part is wrong and when I try to run the image (start a container based on it) I see no port has been exposed.
image

Thank you

@Alexhuszagh
Copy link
Contributor Author

Alexhuszagh commented Nov 5, 2022

@abcd-ca

Do you need CROSS_REMOTE? You should be able to run it natively on your Mac M1, and therefore don't need remote docker, unless you are connecting to the client remotely. You do have to provide the image name itself, however: these aren't named the same so you have to override the image name as follows:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

Does that help? All the image names and tags can be found here.

@abcd-ca
Copy link

abcd-ca commented Nov 5, 2022

@Alexhuszagh Thanks, that does look more like what I need however with that I get,

docker: Error response from daemon: image with reference ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf was found but does not match the specified platform: wanted linux/amd64, actual: linux/arm64.

@Alexhuszagh
Copy link
Contributor Author

Is your local machine the M1? Or are you connecting remotely to it? If so, then you do need CROSS_REMOTE along with the correct connection parameters (SSH connection? TCP?).

@abcd-ca
Copy link

abcd-ca commented Nov 5, 2022

Local is M1. No physically remote build computer yet. Hoping to avoid that

@Alexhuszagh
Copy link
Contributor Author

Alexhuszagh commented Nov 5, 2022

Local is M1. No physically remote build computer yet. Hoping to avoid that

Ah I think I know what the issue is. Sorry for the repeated mentions. You need to do the instructions under target.TARGET.image, or provide them as environment variables.

So the full invocation is:

CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE_TOOLCHAIN="aarch64-unknown-linux-gnu" \
    CROSS_TARGET_ARMV7_UNKNOWN_LINUX_MUSLEABIHF_IMAGE=ahuszagh/aarch64-cross:armv7-unknown-linux-musleabihf \
    cross build --release --target=armv7-unknown-linux-musleabihf

If we deploy native ARM64 images, this should be streamlined a lot later on (we need custom runners which aren't free), but right now it's a bit janky. Does that fix the issue?

@abcd-ca
Copy link

abcd-ca commented Nov 5, 2022

Sorry for the repeated mentions

Are you kidding? Thanks for being so generous with your time and expertise!

Success! 🎉

Here are some test results (docker images already automatically downloaded so time to download those not included):

Build time with default build image
cross clean
time ./build.sh

Finished release [optimized] target(s) in 8m 45s
./build.sh 0.46s user 0.26s system 0% cpu 8:48.76 total

Build time with native aarch64 (M1) build image:
cross clean
time ./build.sh (modified to use your latest env vars)

Finished release [optimized] target(s) in 2m 23s
./build.sh 0.65s user 0.64s system 0% cpu 2:27.52 total

Summary:

About 6.5 minutes (73%) faster!

djkoloski added a commit to rkyv/rkyv that referenced this issue Mar 9, 2023
Cross's android images are causing CI to fail. Maybe related: cross-rs/cross#975
djkoloski added a commit to rkyv/rkyv that referenced this issue Mar 9, 2023
Cross's android images are causing CI to fail. Maybe related: cross-rs/cross#975
@jaysonsantos
Copy link

Hey there folks, are these images being pushed somewhere? In this repo, I can only see linux/amd64 and unknown/unknown
image

@Emilgardis
Copy link
Member

Emilgardis commented Mar 31, 2023

@Alexhuszagh has pushed initial images here as mentioned in this issue.

We have not started implementing this in-org yet, and thus they are not on ghcr.io/cross-rs. You can also build these images yourself

@YOU54F
Copy link
Contributor

YOU54F commented May 30, 2023

Hey @Alexhuszagh

Kudos on the awesome tracking list, I have this saved as a bookmark, was SUPER useful for me, in my current ARMaggedon flex on our open source project.

Linked PR above ☝🏾 to attempt to cover off the case for aarch64-unknown-freebsd

@DoumanAsh

This comment was marked as off-topic.

@CryZe

This comment was marked as off-topic.

@DoumanAsh

This comment was marked as off-topic.

@CryZe

This comment was marked as off-topic.

@Emilgardis

This comment was marked as off-topic.

bors bot added a commit that referenced this issue Sep 14, 2023
1271: feat: add aarch64-unknown-freebsd image r=Emilgardis a=YOU54F

👋🏾 

Thanks for Cross, we've been using it at the pact-foundation to build cross platform targets, and I've gone through the target list and attempted to build for everything I can. See [PR here](pact-foundation/pact-reference#276)

I've managed to build for freebsd in a parallels vm on a mac m1 pro, but makes sense to try and apply that to a Docker image.

Relates to #975

Still requires testing but thought I would drop it here for others to have a look at / test / try out 



Co-authored-by: Yousaf Nabi <you@saf.dev>
@mcandre

This comment was marked as off-topic.

@Emilgardis

This comment was marked as off-topic.

@mcandre
Copy link

mcandre commented Jan 4, 2024

Configuring a DOCKER_DEFAULT_PLATFORM environment variable with value linux/amd64 can smooth out rough edges while awaiting publication of multi-architecture images. You may need to explicitly initialize the Docker buildx subsystem, something that the tug third party utility can do for ya.

https://github.com/mcandre/tug

Recommend to house this configuration in direnv / psenv, in order to reduce conflicts between different software projects you may be working on.

@Coruscant11
Copy link

Coruscant11 commented Jul 30, 2024

Hi @Alexhuszagh !
I was wondering if providing a cross-rs/zig:aarch64 image would make it easier ?

@molenick
Copy link

molenick commented Aug 6, 2024

I'm trying to install libswupdate-dev:arm64 against the default image and have not been able to figure out how using package management.

My cross file looks like this:

[target.aarch64-unknown-linux-gnu]
pre-build = [
    "dpkg --add-architecture $CROSS_DEB_ARCH",
    "apt-get update",
    "apt-get --assume-yes install libswupdate-dev:$CROSS_DEB_ARCH",
]

But when I run:

cross build --target aarch64-unknown-linux-gnu --verbose

I get the following error:

E: Unable to locate package libswupdate-dev:arm64

I've tried a bunch of stuff in the pkg-mngt wrangling but no success. At one point I got it to be located but it wouldn't install because of issues with libc toolchain not being available. I was able to build successfully by directly installing the .deb packages, but my preference is to use apt.

I've read the docs linked from the README a few times but I feel like I'm missing something here about the problem I'm facing.

edit: Is it possible to get command output from the docker setup? I can see the command thats running with export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true but I can't seem to get command output; that make it easier for me to debug.

@Emilgardis
Copy link
Member

edit: Is it possible to get command output from the docker setup? I can see the command thats running with export DEBIAN_FRONTEND=noninteractive DEBCONF_NONINTERACTIVE_SEEN=true but I can't seem to get command output; that make it easier for me to debug.

Yes, simply add -v @molenick

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
A-aarch64-host Area: ARMv8 hosts container-images tracking-issue Issue to track progress for a certain feature/enhancement..
Projects
None yet
Development

No branches or pull requests

10 participants