Emulator-based unit tests no longer work on Travis

[barbeau]

Summary:

There are issues with API Level 23-25 emulators that prevent them from running on Travis.

Here's an example of a failed build:
https://travis-ci.org/googlemaps/android-maps-utils/builds/206977302

Things go unresponsive with the recent master branch (as of 9be09a2). I've tried a number of resolutions in these branches/PRs (for this and another project):

• https://github.com/barbeau/android-maps-utils/tree/test-travis
• Fix #720 - Fix ABI etc for Travis builds OneBusAway/onebusaway-android#721

...but so far no luck.

It seems to boil down to these two problems with the emulators/SDK:

• API Level 23 - https://code.google.com/p/android/issues/detail?id=228113
• API Level 24/25 - https://code.google.com/p/android/issues/detail?id=230083

@stephenmcd or @broady Any chance you can try to get the Android emulator/SDK team to take a look at the above? I'm not very familiar with how this is set up, but it may be a simple issue with the metadata associated with the SDK when fetched by something like Travis.

For now, as a workaround in PR #370, I've changed it to simply build the project, without spinning up an emulator and running the unit tests. This is obviously a huge inconvenience, as you still need to run the unit tests locally to verify that they pass.

Steps to reproduce:

Run a Travis build as of 9be09a2 in the master branch, with the .travis.yml config shown in https://github.com/googlemaps/android-maps-utils/blob/8f7d4b153d7281c818853474361914d3aefc0678/.travis.yml.

Expected behavior:

Emulator image should be downloaded, started, and the unit tests should run via ./gradlew clean check connectedCheck -x library:signArchives -PdisablePreDex.

Observed behavior:

The emulator times out (or there is other behavior with missing ABIs, etc. depending on the API Level) - see https://travis-ci.org/googlemaps/android-maps-utils/builds/206977302 for an example, which shows:

5.61s$ echo no | android create avd --force -n test -t "android-"$ANDROID_API_LEVEL --abi $ANDROID_ABI --tag $ANDROID_TAG
Android 7.0 is a basic Android platform.
Do you wish to create a custom hardware profile [no]Created AVD 'test' based on Android 7.0, Google apis ARM (armeabi-v7a) processor,
with the following hardware config:
hw.lcd.density=240
hw.ramSize=512
vm.heapSize=48
before_script.2
0.00s$ emulator -avd test -no-skin -no-window &
$ android-wait-for-emulator
emulator: WARNING: the -no-skin flag is obsolete. to have a non-skinned virtual device, create one through the AVD manager
Creating filesystem with parameters:
    Size: 69206016
    Block size: 4096
    Blocks per group: 32768
    Inodes per group: 4224
    Inode size: 256
    Journal blocks: 1024
    Label: 
    Blocks: 16896
    Block groups: 1
    Reserved block group size: 7
Created filesystem with 11/4224 inodes and 1302/16896 blocks
pulseaudio: pa_context_connect() failed
pulseaudio: Reason: Connection refused
pulseaudio: Failed to initialize PA contextaudio: Could not init `pa' audio driver
emulator: Listening for console connections on port: 5554
emulator: Serial number of this emulator (for ADB): emulator-5554
[140504510871296]:WARNING:./android/base/files/IniFile.cpp:158:Failed to process .ini file /home/travis/.android/emu-update-last-check.ini for reading.
Your emulator is out of date, please update by launching Android Studio:
 - Start Android Studio
 - Select menu "Tools > Android > SDK Manager"
 - Click "SDK Tools" tab
 - Check "Android SDK Tools" checkbox
 - Click "OK"
[140503899166464]:WARNING:./android/base/files/IniFile.cpp:158:Failed to process .ini file /home/travis/.android/emu-update-last-check.ini for reading.
Waiting for emulator to start
Waiting for emulator to start
Waiting for emulator to start
Waiting for emulator to start
Waiting for emulator to start
Waiting for emulator to start
Waiting for emulator to start
Waiting for emulator to start
No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
Check the details on how to adjust your build configuration on: https://docs.travis-ci.com/user/common-build-problems/#Build-times-out-because-no-output-was-received
The build has been terminated

Device and Android version:

Above example is using API Level 23 emulator - see https://github.com/googlemaps/android-maps-utils/blob/8f7d4b153d7281c818853474361914d3aefc0678/.travis.yml for exact config.

Screenshots:

N/A

[shawnshaligram]

so the temp workaround for this is to use an API Level < 23 ?

[barbeau]

I had different issues with API 22 - see OneBusAway/onebusaway-android#720 (comment). There is supposedly a fix now for newer APIs (see OneBusAway/onebusaway-android#720 (comment)) although I haven't had a chance to try it myself yet.

[gengjiawen]

Run into similar issue in travis ci and circle ci. I reported to https://issuetracker.google.com/issues/76200998.

[mmcc007]

The system-images;android-22;default;armeabi-v7a with android-28 tools combo worked for me on travis. I was able to get Flutter integration tests to pass (on both linux and osx). Should work for other integration tests too. Here's a test of several combos for emulator I tried to startup:
https://travis-ci.org/mmcc007/test_emulators

[barbeau]

This configuration change recently started working for GPSTest - we can give it a shot - barbeau/gpstest@e4d77bf.

[mochadwi]

This configuration change recently started working for GPSTest - we can give it a shot - barbeau/gpstest@e4d77bf.

Does this also works for API above 18? @barbeau

[barbeau]

@mochadwi I don't know, I've only tested it on the API 18 emulator. Let me know if you try it!

[mmcc007]

@mochadwi system-images;android-22;default;armeabi-v7a with android-28 tools has been working consistently (although slooowly) for almost a year now on this project:
https://github.com/brianegan/flutter_architecture_samples
For recent build see:
https://travis-ci.org/brianegan/flutter_architecture_samples/jobs/540887819#L1234

[mochadwi]

@mochadwi system-images;android-22;default;armeabi-v7a with android-28 tools has been working consistently (although slooowly) for almost a year now on this project:
https://github.com/brianegan/flutter_architecture_samples
For recent build see:
https://travis-ci.org/brianegan/flutter_architecture_samples/jobs/540887819#L1234

Awesome. After carefully considering the build speed on Travis, we choose using android-16 instead

[mmcc007]

@mochadwi Sounds good. Depending on the complexity of your tests, emulator startup is one thing, and emulator performance is another. I have not tried to benchmark emulator performance. But here is my equivalent benchmark on startup time.
https://travis-ci.org/mmcc007/test_emulators/builds/495210049

[rcgonzalezf]

The android emulator api 23 is working for me, I'm leaving a link to a successful build:
https://travis-ci.org/github/rcgonzalezf/weather-app-demo/builds/704834288

After reading this thread, I changed it to api 22. Just wanted to mention that is working.