arm64 / aarch64 support for Fedora CoreOS

[vielmetti]

Reading through the channel minutes for the last meeting, @glevand mentioned an interest in first-class arm64 / aarch64 support for Fedora CoreOS, a sentiment which I share.

I'd like to add this to the next agenda (Wednesday of week 29, July 18 2018) and start to identify the current state of the components that would go into this, look for some target hardware to point at first, and see what infrastructure needs there are.

[vielmetti]

The meeting the week 29 was cancelled, so the next time is W30, July 25, 2018.

I will be traveling during the meeting time, but hope that I will have wifi that will work.

The mailing list includes discussion of "ore" a provisioning tool so there's some chance that might be relevant to this as well.

[sinnykumari]

In today's meeting, we agreed upon keeping me (Sinny), Geoff and ed-packet as the point of contact for aarch64 considerations during FCOS design decisions.

[glevand]

In the meeting today we discussed how FCOS for arm64 would be built, and the outcome was that the fedora build system does not do cross compilation. The arm64 builds will need to be done in an arm64 execution environment.

[vielmetti]

Terrific. If someone, perhaps @glevand or @sinnykumari , would like to fill out a request at https://github.com/worksonarm/cluster/issues/new I can get some arm64 hardware available for the task.

[vielmetti]

From the meeting today:

• ACTION: sanja to create docs repo this week (jbrooks, 16:42:29)

@sanjabonic I think that's you, but I'm not 1000% certain.

[dustymabe]

@vielmetti - is that related to this issue for arm64 / aarch64 support?

[vielmetti]

@dustymabe I believe so, this is an excerpt from this week's meeting notes.

[glevand]

The action item to create a docs repo is so ajeddeloh can have a place for his rolling design doc. It is not related to arm64.

[ajeddeloh]

Removing the meeting tag since there hasn't been discussion about it in the meetings. Feel free to re-add if anyone wants to bring something up about this next meeting.

[sinnykumari]

@vielmetti It took me long time but have finally filed a ticket for getting access to aarch64 hardware https://github.com/WorksOnArm/cluster/issues/110 . Thanks!

[vielmetti]

@sinnykumari account is now set up! thanks

[sinnykumari]

Thanks Ed!

[agners]

What is the conclusion on this ticket? Are there arm64 builds publicly available today?

[dustymabe]

@agners - we don't have any arm64/aarch64 builds being output from the pipeline today, but we do believe it works if you build locally using coreos-assembler. The limitation right now is mostly on our build side (i.e. our build environment doesn't support aarch64). Here is a comment stating where we currently are on that front: #262 (comment)

[vielmetti]

Thanks @dustymabe - let me explore with you how to address the aarch64 build environment with the @WorksOnArm resources in a test and prototyping phase, with a goal to have a timeline to have a fully supported Fedora CoreOS hardware/infrastructure rather than assume long-term use of the sponsored @WorksOnArm resources.

[barnscott]

@vielmetti has there been any update with the arm build environment?

I'm interested in coreos support for Rockpro64 and was wondering if there is an arm image available.
I think the board gets mainline support on kernel 5.7, but may be mistaken, so I'm hoping FCOS 33 would support Rockpro64 with the assumption the there is an arm image available.
https://www.pine64.org/rockpro64/

[vielmetti]

@barnscott - the current Packet and @WorksOnArm environment has Ampere servers available, either on an on-demand or reserved commercial basis or sponsored and subsidized by Arm through the Works on Arm program.

[barnscott]

@vielmetti @dustymabe Thank you. Is there a plan/timeline for general availability of an ARM image of CoreOS?

[dustymabe]

Unfortunately we don't currently have a timeline.

[kevinzs2048]

Hi, this is Kevin from Linaro.
I wonder if we can help to make than happen by porting or rebuilding this OS co-work with Fedora CoreOS team.
We'd like to know if there is a guide or steps of this huge job so that we can take step to do it.

[jomeier]

@dustymabe
Could you provide all necessary resources (links, tools pipelines, …) that are involved in doing that?

Maybe the community can help in making a PoC for that.

FCOS on the Raspberry … how cool is that :-) !?

[nullr0ute]

FCOS on the Raspberry … how cool is that :-) !?

Not very TBH, but on aarch64 as a whole.... Most of the core bits support aarch64 just fine, we already support it on Fedora IoT.

[barnscott]

@nullr0ute
I agree, and I think IoT on Raspberry Pi is a good fit. The only advantage I can think of from FCOS may be automatic updates. Rebasing versions may be a particular pain point that doesn't affect FCOS since it is rolling.

As far as FCOS on arm, that is a bigger conversation. Cloud providers are beefing up ARM compute availability, so someone may have specialized workloads that benefit from FCOS on arm.

[glevand]

Just to mention it, @Madhu2264 and I did some work on coreos-assembler a while ago to allow it to use qemu system emulation so that you can build and run arm64 binaries on other architectures. With such support I think it would be relatively straight forward to add arm64 cross builds to existing build systems that are doing amd64 builds.

[jlebon]

Cross-posting: https://lists.fedoraproject.org/archives/list/coreos@lists.fedoraproject.org/thread/CLZZE3UKOTLDQQRMDC2Z4MKBOIND64BO/

Just to mention it, @Madhu2264 and I did some work on coreos-assembler a while ago to allow it to use qemu system emulation so that you can build and run arm64 binaries on other architectures.

Interesting. What kind of speeds were you getting compared to native? The other thing too is that it's not just building; we also really want to be able to test the artifacts.

[kevinzs2048]

Share some info. Thanks @glevand @Madhu2264, I have successfully built the qemu and OpenStack images on Arm64. I've used it into our internal system test now and will quickly upload to Linaro Developer Cloud(www.linaro.cloud) as the Kubernetes as a service base distro.

[glevand]

@jlebon, of course there is some overhead for the emulation. We used system emulation, but I think it would be possible to also use user mode emulation, which would have less overhead, but would need more rework of coreos-assembler to support.

As for testing, that should just work. A qemu VM is just a VM, so interaction with it is essentially the same as a KVM VM or a remote machine.

[cgwalters]

I think @jcajka is working on keeping things going here, see e.g.: coreos/coreos-assembler#1448

So what we're mostly blocked on is #262

[jcajka]

I don't think that non-native is the way to go. It has it own quirks, bugs and overheads. It has been Fedora's approach to use native(HW/VM) for "production" builds. IMO HW emulation has only place in HW bring-up when "physical" HW is not yet available/"common"(IIRC RISC-V folks have been using qemu for initial round of builds of Fedora, they switched to boards(HiFive?) as they became available).

I'm currently looking in to way how to enable the pipeline, without the need for full openshift/kube clusters(per architecture) as a "builders" there. My understanding is that infra doesn't have enough resources(both manpower and HW) to support them for any non-x86_64 architecture.

Native VM/host(they should be available in the Fedora/CentOS infra after the DC move is finished) as jenkins builders would be acceptable by Infra and FCOS team. Currently I'm diving in the code of the FCOS pipeline to see how to make that happen(while keeping the builds that get to the https://fedorapeople.org/groups/fcos-images/ relatively stable, btw pushes there are not yet automated, and fixing tests failures).

Also I think that rock64pro/rbp4(I have them both :)) and other aarch64 sbcs that AFAIK don't have support for the standard booting(UEFI/EBBR?, yet?) are tasks on it's own(building special images for them while retaining all the features of the FCOS, updates,...).

As a bit side note if you have aarch64 sbc (rpi4 running Fedora rawhide in my case), you can build FCOS from scratch in ~1h(disk/sdcard speed being the biggest bottleneck for me, on 20Mbit/s internet connectivity) just following the cosa readme and run it in VM there for hacking/devel/fun.

[rhatdan]

What is the latest on this? @baude is working on podman for MAC and is working with Fedora CoreOS for his Container OS, but he needs it for Intel and Arm versions of MAC. Is this supported?

[bgilbert]

It's not officially supported currently.

[dustymabe]

Update.. We are currently building aarch64 artifacts in our devel pipelines (i.e. testing-devel) and hope to release the content in the next several weeks.

[dustymabe]

This week's releases all have aarch64 artifacts. We're working on getting our official website updated to reflect the new architecture. In the meantime you can find aarch64 artifacts in our unofficial release browser.

[dustymabe]

The fix for this went into stable stream release 34.20210821.3.0.

[mkumatag]

This week's releases all have aarch64 artifacts. We're working on getting our official website updated to reflect the new architecture. In the meantime you can find aarch64 artifacts in our unofficial release browser.

@dustymabe any plans for ppc64le platform? :)

[travier]

@dustymabe any plans for ppc64le platform? :)

Please file a distinct issue for tracking.