Skip to content

Commit

Permalink
[tests] Check for emulator issues and error early
Browse files Browse the repository at this point in the history
The Android emulator is a "wonderful" beast. Case in point:
[PR Build #1074][0], which *hung* during emulator startup, for
*5 hours*:

	09:13:49 Hax is enabled
	09:13:49 Hax ram_size 0x0
	09:13:49 HAX is working and emulator runs in fast virt mode.
	...
	09:13:56 		adb I 06-21 09:13:56 47148 21294922 adb_io.cpp:75] readx: fd=3 wanted=4
	...
	# 5.5 hours later...
	14:47:17 		Tool /Users/builder/android-toolchain/sdk/platform-tools/adb execution finished.

It "finished" because I manually killed the Job, as it had take over
6 *hours*, cumulative, to do nothing.

(*Arguably* we shouldn't have a 10 hour timeout on Jenkins jobs, but
Mono bumps can take upwards of 6 hours to build, so a 6 hour time
isn't necessarily unusual...)

This also isn't our first attempt at improving emulator reliability.
See also commits c089267, bc6440b, 3fa9e9e, b54f8cd, 3b893cd,
7450efc, and 6358a64. (Is that enough? Likely not. Is that all of
them? Probably not.)

Take yet another stab at improving things: in this case, improving
error checking, so that *when* an error occurs we can *fail early*, as
opposed to waiting for hours on end for a Jenkins timeout to occur
*or* for me to get annoyed at general Jenkins build slowness enough to
track down the hung job and manually kill it...

Update the `emulator` process launch so that we have event handlers
for stdout and stderr messages, so that we can look for currently
known error conditions, which fall into two categories:

 1. HAXM needs reinstallation, or
 2. Another VM is in use.

HAXM appears to need reinstallation when no memory is allocated:

	Hax ram_size 0x0

I don't know *why* HAXM would require *re*-installation in these
circumstances, but that generally appears to fix the problem.

When another VM is in use, e.g. when I'm running a Veertu VM on macOS,
`emulator` also fails to launch, writing the following to stderr:

	Failed to sync vcpu reg
	Failed to sync HAX vcpu contextInternal error: Initial hax sync failed

If either of these conditions occur, the `<StartAndroidEmulator/>`
task should *fail*, with a (hopefully) relevant error message.

With these changes in place, remove the `<Sleep/>` task invocations
from the `AcquireAndroidTarget` target, as the new 20 *second* timeout
in `<StartAndroidEmulator/>` removes the need for the first Sleep, and
the 2nd sleep doesn't appear to do any good.

Followup to c089267!

> in case the added sleep doesn't help, we might at least get some
> more useful information

We do get more information. It's not entirely *useful*: `adb` is
waiting for `adbd` to provide it information, and that information
never arrives:

	adb I 06-21 09:13:56 47148 21294922 adb_io.cpp:75] readx: fd=3 wanted=4

...and there it waits, seemingly forever, until Job timeout or Job
death occurs.

[0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android-pr-builder/1074/
  • Loading branch information
jonpryor authored and radekdoulik committed Jun 22, 2017
1 parent 74f1bd6 commit db668ba
Show file tree
Hide file tree
Showing 2 changed files with 66 additions and 8 deletions.
8 changes: 0 additions & 8 deletions build-tools/scripts/UnitTestApks.targets
Original file line number Diff line number Diff line change
Expand Up @@ -42,21 +42,13 @@
<Output TaskParameter="AdbTarget" PropertyName="_EmuTarget" />
<Output TaskParameter="EmulatorProcessId" PropertyName="_EmuPid" />
</StartAndroidEmulator>
<Sleep
Condition=" '$(_ValidAdbTarget)' != 'True' "
Milliseconds="10000"
/>
<Xamarin.Android.Tools.BootstrapTasks.Adb
EnvironmentVariables="ADB_TRACE=all"
Condition=" '$(_ValidAdbTarget)' != 'True' "
Arguments="$(_AdbTarget) wait-for-device"
ToolExe="$(AdbToolExe)"
ToolPath="$(AdbToolPath)"
/>
<Sleep
Condition=" '$(_ValidAdbTarget)' != 'True' "
Milliseconds="1000"
/>
<Xamarin.Android.Tools.BootstrapTasks.Adb
EnvironmentVariables="ADB_TRACE=all"
Condition=" '$(_ValidAdbTarget)' != 'True' "
Expand Down
Original file line number Diff line number Diff line change
@@ -1,6 +1,7 @@
using System;
using System.Diagnostics;
using System.IO;
using System.Threading;
using Microsoft.Build.Framework;
using Microsoft.Build.Utilities;

Expand Down Expand Up @@ -68,14 +69,79 @@ void Run (string emulator)
Arguments = arguments,
UseShellExecute = false,
CreateNoWindow = true,
RedirectStandardOutput = true,
RedirectStandardError = true,
WindowStyle = ProcessWindowStyle.Hidden,
};

Log.LogMessage (MessageImportance.Low, $"Environment variables being passed to the tool:");
var p = new Process () {
StartInfo = psi,
};

var sawError = new AutoResetEvent (false);

DataReceivedEventHandler output = null;
output = (o, e) => {
Log.LogMessage (MessageImportance.Low, $"[emulator stdout] {e.Data}");
if (string.IsNullOrWhiteSpace (e.Data))
return;
if (e.Data.StartsWith ("Hax ram_size", StringComparison.Ordinal) &&
e.Data.EndsWith (" 0x0", StringComparison.Ordinal)) {
Log.LogError ("Emulator failed to start: ram_size is 0MB! Please re-install HAXM.");
sawError.Set ();
}
};
DataReceivedEventHandler error = null;
error = (o, e) => {
Log.LogMessage (MessageImportance.Low, $"[emulator stderr] {e.Data}");
if (string.IsNullOrWhiteSpace (e.Data))
return;
if (e.Data.StartsWith ("Failed to sync", StringComparison.Ordinal) ||
e.Data.Contains ("Internal error")) {
Log.LogError ($"Emulator failed to start: {e.Data}");
Log.LogError ($"Do you have another VM running on the machine? If so, please try exiting the VM and try again.");
sawError.Set ();
}
};

p.OutputDataReceived += output;
p.ErrorDataReceived += error;

p.Start ();
p.BeginOutputReadLine ();
p.BeginErrorReadLine ();

const int Timeout = 20*1000;
int i = WaitHandle.WaitAny (new[]{sawError}, millisecondsTimeout: Timeout);
if (i == 0 || Log.HasLoggedErrors) {
p.Kill ();
return;
}

p.CancelOutputRead ();
p.CancelErrorRead ();

p.OutputDataReceived -= output;
p.ErrorDataReceived -= error;

p.OutputDataReceived += WriteProcessOutputMessage;
p.ErrorDataReceived += WriteProcessErrorMessage;

p.BeginOutputReadLine ();
p.BeginErrorReadLine ();

EmulatorProcessId = p.Id;
}

static void WriteProcessOutputMessage (object sender, DataReceivedEventArgs e)
{
Console.WriteLine (e.Data);
}

static void WriteProcessErrorMessage (object sender, DataReceivedEventArgs e)
{
Console.Error.WriteLine (e.Data);
}
}
}

0 comments on commit db668ba

Please sign in to comment.