Unable to use Fast Mode 2 Intel-on-ARM with Ubuntu 24.04+

[smbambling]

I'm unable to determine the cause, but when moving from Ubuntu 22.04 to 24.04 I'm no longer able to run x64 containers on a K3s instance running within a Lima Ubuntu VM.

The only change I made to my Lima template was updating the image. The version of K3s and configuration setting remain the same.

Does anyone have an example template where they are able to successfully run intel x64 containers on an Ubuntu 24+

[jandubois]

Please attach a template to repro the issue!

[afbjorklund]

In this case, the important information was the host info (macOS 13) and the vm info (Ubuntu 24.04) - and the error

rosetta error: unhandled auxillary vector type 28

There are known issues when running new Linux kernels on old macOS versions, but was only reported to colima 😔

[smbambling]

I had a lengthly chart with Anders Björklund this morning who was able to fill in some knowledge gaps and help determine the root cause.

This only looks to be an issues with MacOS 13.x

This error is due to to two additional auxiliary vectors added in >torvalds/linux@317c819. As a short-term workaround, the best option is to stay on Linux kernel versions older than 6.3 or set CONFIG_RSEQ=n when compiling your kernel.

Lima uses "OCF", but on the old system it should be "POCF" to preserve.

References:

• Crash with rosetta in 0.6.0 (regression from 0.5.6) abiosoft/colima#855
• Rosetta results with error for any AMD64 binary utmapp/UTM#5567
• https://developer.apple.com/documentation/virtualization/running_intel_binaries_in_linux_vms_with_rosetta
• Crash with rosetta in 0.6.0 (regression from 0.5.6) abiosoft/colima#855 (comment)

[afbjorklund]

Lima uses "OCF", but on the old system it should be "POCF" to preserve.

As noted in those issues, the Apple recommendation is the exact opposite:

"Important: When using Rosetta in macOS 13, set the preserve option to no."

So we still don't what fix is needed, except for upgrading your OS of course

[smbambling]

I'm happy to mark this as resolved and I think we have a fair bit of information in here that should hopefully help others if they stumble upon a similar issue.

[afbjorklund]

As far as I know, it is still broken on macOS 13. But I'm not planning on downgrading.

[smbambling]

I'm happy to aid in some testing, just need some details methods / instructions.

[afbjorklund]

The only theory so far is that Apple got it backwards, and that there should be a "P" added on macOS 13:

https://github.com/lima-vm/lima/blob/v1.0.0-beta.0/pkg/cidata/cidata.TEMPLATE.d/boot/05-rosetta-volume.sh#L17

But since you also mentioned that editing /usr/lib/binfmt.d/rosetta.conf doesn't help, I'm not sure...

It seems to be working fine (with Ubuntu 24.04) on macOS 14.7 - without the recommended "preserve=yes"