Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

fix emulator OutOfMemory by forcing zygote init race win via warm reboot #12113

Merged
merged 1 commit into from
Aug 20, 2022

Conversation

mikehardy
Copy link
Member

Pull Request template

Purpose / Description

Our emulator tests have all the sudden become very flaky.
Proximate cause appears to be OutOfMemory death spiral in emulator processes.
Root cause is hypothesized to be a race between correct (sufficient) memory sizing and zygote init in boot sequence
This should work around that

Details here #11091 (comment)

Fixes

Unlogged general flakiness in tests_emulator.yml

Approach

Warm reboot the emulator by restarting the zygote process
The sleep 5 / sleep 10 are just rough guesses that a) time to settle is needed b) that time is sufficient

How Has This Been Tested?

It is in testing, adb logcat should show a zygote with something other than -Xmx16m
Ideally we see a -Xmx16m at start then a -Xmx"not 16m" later in zygote / art init arguments
It may take a couple runs (my guess is 3?) before the race is initially lost / then repaired to prove this works

Learning (optional, can help others)

I found this: https://support.circleci.com/hc/en-us/articles/360021812453-Common-Android-memory-issues#zygote

Which leads to this: https://www.rainforestqa.com/blog/hunting-race-condition-in-android-10-emulator

Checklist

Please, go through these checks before submitting the PR.

  • You have a descriptive commit message with a short title (first line, max 50 chars).
  • You have commented your code, particularly in hard-to-understand areas
  • You have performed a self-review of your own code
  • UI changes: include screenshots of all affected screens (in particular showing any new or changed strings)
  • UI Changes: You have tested your change using the Google Accessibility Scanner

@mikehardy
Copy link
Member Author

First run:


08-20 11:28:14.017  1694  1694 I zygote64: option[0]=-Xzygote
08-20 11:28:14.017  1694  1694 I zygote64: option[1]=exit
08-20 11:28:14.017  1694  1694 I zygote64: option[2]=vfprintf
08-20 11:28:14.017  1694  1694 I zygote64: option[3]=sensitiveThread
08-20 11:28:14.017  1694  1694 I zygote64: option[4]=-verbose:gc
08-20 11:28:14.017  1694  1694 I zygote64: option[5]=-Xms4m
08-20 11:28:14.017  1694  1694 I zygote64: option[6]=-Xmx16m

and a bunch of GC memory trauma, then behold! The hot restart hits, processes die and I see


08-20 15:29:08.980  3280  3280 I zygote64: option[0]=-Xzygote
08-20 15:29:08.981  3280  3280 I zygote64: option[1]=exit
08-20 15:29:08.981  3280  3280 I zygote64: option[2]=vfprintf
08-20 15:29:08.981  3280  3280 I zygote64: option[3]=sensitiveThread
08-20 15:29:08.981  3280  3280 I zygote64: option[4]=-verbose:gc
08-20 15:29:08.981  3280  3280 I zygote64: option[5]=-Xms4m
08-20 15:29:08.981  3280  3280 I zygote64: option[6]=-Xmx512m

it works! with testing proof on the first time. 🎯

@mikehardy mikehardy merged commit f59fe0a into ankidroid:main Aug 20, 2022
@mikehardy mikehardy deleted the emulator-zygote-race-workaround branch August 20, 2022 15:47
@github-actions github-actions bot added this to the 2.16 release milestone Aug 20, 2022
@mikehardy mikehardy mentioned this pull request Aug 20, 2022
@david-allison
Copy link
Member

Incredible work, thanks!!!!

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants