Hardware acceleration

[mattjohnsonpint]

This is a bit of continuation from #279. (@mrk-han especially)

I've been doing some experimenting, and I don't think thatinstalling HAXM on MacOS under GitHub Actions hosted runners is actually doing anything whatsoever for the Android emulator.

brew install --cask intel-haxm does indeed say it succeeds, but it doesn't actually work. Same with sudo $ANDROID_HOME/cmdline-tools/latest/bin/sdkmanager --install "extras;intel;Hardware_Accelerated_Execution_Manager" which does the same thing without brew.

• The HAXM docs say that you should see the HAXM kernel extension running with kextstat | grep intel, but that returns nothing. Run without grep shows other extensions, but not HAXM.

• The android docs here say that on macOS 10.10 and higher, the built-in Hypervisor.Framework is used first, and HAXM is a fallback. It also says here that on macOS 10.13 and higher one has to enable security permissions manually in the system preferences UI in order to load a third-party (non-apple) kernel extension at all.

This aligns with my findings. After installing HAXM (with either brew or sdkmanager), /Library/Extensions/intelhaxm.kext is on disk, but not loaded. sudo kextload /Library/Extensions/intelhaxm.kext gives an error:

Executing: /usr/bin/kmutil load -p /Library/Extensions/intelhaxm.kext
Error Domain=KMErrorDomain Code=27 "Extension with identifiers com.apple.nke.rvi,com.intel.kext.intelhaxm not approved to load. Please approve using System Preferences."
UserInfo={NSLocalizedDescription=Extension with identifiers com.apple.nke.rvi,com.intel.kext.intelhaxm not approved to load. Please approve using System Preferences.}
Error: Process completed with exit code 27.

And regardless of whether I install HAXM or not, checking the emulator with $ANDROID_HOME/emulator/emulator -accel-check always gives the same result:

accel:
0
Hypervisor.Framework OS X Version 12.
accel

I interpret this to mean that the Android emulator is indeed using hardware acceleration, as provided by Hypervisor.Framework, and I cannot use HAXM because I don't have ability to set the security permission programatically.

Am I wrong?

(I had the same results on macos-11 or macos-12 GitHub Actions hosted runners)

CC: @vaind (any thoughts?)

[vaind]

While I haven't done such a thorough investigation I've had my suspicion this is the case, too.

Also, I have a screen recording from CI that asks user to confirm the extension:

[mrk-han]

@mattjohnsonpint I'm really glad you started this issue, because I'm working through the same thing now. The pop-up is blocking a lot, and I'm not sure it's achieving VM acceleration.

The only difference I'm running into, is that I thought I was not getting a valid accel-check before running the command -- I will try to make a bare-bones repo and test this

[mrk-han]

I have been talking back and forth while trying to figure this stuff out with @igorboskovic3 who sent me https://github.com/igorboskovic3/actions-test/actions/runs/3228474831 which also has a system-extension updated popup. Screenshot of Popup

I really do think we need more clarity on this from the runner team.

Here are the few issues I have been communicating in related to HAXM:
Update/Add HAXM to macOS 11 and macOS 12 agents

Android emulator boots up with black screen on microsoft hosted agent with mac-11 and Mac-12 in Azure pipeline

Getting spammed with [CAMetalLayer nextDrawable] returning nil because device is nil on latest macOS Agent

Another thing I want to look into is potentially using the new SKIA Rendering $ANDROID_HOME/platform-tools/adb shell 'su;setprop debug.hwui.renderer skiagl;stop;start' -- but it requires the emulator to be rooted I believe.

[tenzap]

I just tried locally on an image with gpu accel disabled:

adb shell "su root setprop debug.hwui.renderer skiagl"
adb shell "su root stop"
adb shell "su root start"

The UI didn't seem much more responsive than without skiagl

[mrk-han]

@tenzap This checks out. Locally our machines are much more powerful nowadays compared to CI, but the main difference is on CI we are running on VMs so it's nested virtualization with the emulator vm. Locally we obviously don't have that GPU passthrough constraint.

[mattjohnsonpint]

The main reason I am digging on this is that some of our Android tests are extremely flaky - but only when running in Github Actions. Run the same tests on an emulator on either my Mac or Windows machines and they pass every time. Even if I let it run repeatedly for hours.

Also sometimes the emulators in CI flat out fail to boot.

Also API 32 seems more flaky than others - not sure why.

Originally I thought perhaps the issue was because the warning we get:

 WARNING | Running on a system with less than 6 logical cores. Setting number of virtual cores to 1

But like I said, it always works locally - even on an AVD configured for a single core.

So just process of elimination - trying to figure out what is special about the CI environment that causes the flakiness. Perhaps it's hardware acceleration, perhaps not.

[tenzap]

API32 & 33 are very flaky for me too. And there are no problems locally. Howerver if locally I disable gpu accel, I notice that the UI is way much slower.

[tenzap]

Maybe you are interested in this:

"macOS emulator is super slow on machines with less than 6 logical cores."

https://android.googlesource.com/platform/external/qemu/+/refs/heads/emu-master-dev/android-qemu2-glue/main.cpp#2519

#ifdef __APPLE__
        // macOS emulator is super slow on machines with less
        // than 6 logical cores.
        if (System::get()->getCpuCoreCount() < 6) {
            dwarning(
                    "Running on a system with less than 6 logical cores. "
                    "Setting number of virtual cores to 1");
            hw->hw_cpu_ncore = 1;
        }
#endif

[mrk-han]

@mattjohnsonpint Same here. We have quite a few teams all trying to run affordable automation leveraging Github Actions + Github Action Runners after migrating from a self-hosted solution (KVM enabled Ubuntu Agent hosted on MacStadium)

I have also been super skeptical of why WARNING | Running on a system with less than 6 logical cores. Setting number of virtual cores to 1 is happening too.

And yes, I'm pretty sure I could run the test suite with 4x sharding locally on our M1/M2 MacBooks, so it's quite hilarious we're in this state on CI.

[mrk-han]

@mattjohnsonpint I'm starting to think there's much of an advantage at all with running on macOS agents. We used to run on them because HAXM was supported, and things were more "stable" on macOS than Ubuntu Agents (The standard Github ones)

If we're going to have the same performance, I'd rather run on 1/10th the cost Ubuntu agents than the macOS agents.

Also, the Ubuntu agents, if you have access to the larger agent beta, can provide way more powerful machines.

[mrk-han]

@mattjohnsonpint I just did a test and discovered that if you run emulator -accel-check with the new tools, e.g. $ANDROID_HOME/emulator/emulator -accel-check then it always passes, but if you run emulator -accel check with the old tools, then it is fails, e.g. $ANDROID_HOME/tools/emulator

This is more than confusing to me.

https://github.com/mrk-han/Emulator-Sandbox/actions/runs/3374542418

New Tools -> "$ANDROID_HOME"/emulator/emulator -accel-check

accel:
0
Hypervisor.Framework OS X Version 12.6
accel

Old Tools -> "$ANDROID_HOME"/tools/emulator -accel-check

accel:
6
HAXM is not installed on this machine
accel

[tenzap]

Maybe an alternative would be to use an Automated Test Device (ATD) image
https://developer.android.com/studio/releases/platforms#12

Only available currently on API30 & API31 (canary channel) though.

./cmdline-tools/latest/bin/sdkmanager --install "system-images;android-30;google_atd;x86"
./cmdline-tools/latest/bin/sdkmanager --channel=3 --install "system-images;android-31;google_atd;x86_64"

[ychescale9]

this
shows how to run instrumented tests with Gradle managed device on GitHub actions.

[tenzap]

Thanks. I was mainly concerned here by the slowlyness & flakiness of images for recent apis

[ychescale9]

That's not something a tool like this can fix unfortunately.

[mrk-han]

Ahh finally the ATD images for Android 12 are out! Good to know.

I did not have much success with the Android-30 ATD images on Github's macOS Runner.

[mrk-han]

See this comment: #46 (comment)