Skip to content

Commit

Permalink
Add --signal-app-end to detect test run end for iOS/tvOS 14+ devices (
Browse files Browse the repository at this point in the history 
#643)

Signals the test runner to log a given tag at the end of the test run and ends the test run. Useful for iOS/tvOS 14+ where mlaunch cannot detect the app quit.

This functionality should be eventually superseded by a full protocol between the app and XHarness but this way keeps it backwards compatible and also won't break Xamarin who is sharing some of the code.
  • Loading branch information
@premun
premun committed Jun 18, 2021
1 parent 07da2f5 commit c1f8287
Show file tree
Hide file tree
Showing 30 changed files with 826 additions and 182 deletions.
25 changes: 25 additions & 0 deletions README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -180,6 +180,31 @@ To run the E2E tests, you can find a script in `tools/` that will build everythi
./tools/run-e2e-test.sh Apple/Simulator.Tests.proj
```

## Troubleshooting

Some XHarness commands only work in some scenarios and it's good to know what to expect from the tool.
Some Android/Apple versions also require some workarounds and those are also good to know about.

### My Apple unit tests are not running

For the `apple test` command, XHarness expects the application to contain a `TestRunner` which is a library you can find in this repository.
This library executes unit tests similarly how you would execute them on other platforms.
However, the `TestRunner` from this repository contains more mechanisms that help to work around some issues (mostly in Apple platforms).

The way it works is that XHarness usually sets some [environmental variables](https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.iOS.Shared/Execution/EnviromentVariables.cs) for the application and the [`TestRunner` recognizes them](https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.TestRunners.Common/ApplicationOptions.cs) and acts upon them.

The workarounds we talk about are for example some TCP connections between the app and XHarness so that we can stream back the test results.

For these reasons, the `test` command won't just work with any app. For those scenarios, use the `apple run` commands.

### iOS/tvOS device runs are timing out

For iOS/tvOS 14+, we have problems detecting when the application exits on the real device (simulators work fine).
The workaround we went with lies in sharing a random string with the application using an [environmental variable `APP_END_TAG`](https://github.com/dotnet/xharness/blob/main/src/Microsoft.DotNet.XHarness.iOS.Shared/Execution/EnviromentVariables.cs) and expecting the app to output this string at the end of its run.

To turn this workaround on, run XHarness with `--signal-app-end` and make sure your application logs the string it reads from the env variable.
Using the `TestRunner` from this repository will automatically give you this functionality.

## Contribution

We welcome contributions! Please follow the [Code of Conduct](CODE_OF_CONDUCT.md).
Expand Down
190 changes: 112 additions & 78 deletions src/Microsoft.DotNet.XHarness.Apple/AppOperations/AppRunner.cs
View file
Original file line number Diff line number Diff line change
Expand Up @@ -42,7 +42,7 @@ public AppRunner(
ILogs logs,
IHelpers helpers,
Action<string>? logCallback = null)
: base(processManager, captureLogFactory, logs, mainLog, logCallback)
: base(processManager, captureLogFactory, logs, mainLog, helpers, logCallback)
{
_processManager = processManager ?? throw new ArgumentNullException(nameof(processManager));
_snapshotReporterFactory = snapshotReporterFactory ?? throw new ArgumentNullException(nameof(snapshotReporterFactory));
Expand All @@ -56,16 +56,33 @@ public AppRunner(
public async Task<ProcessExecutionResult> RunMacCatalystApp(
AppBundleInformation appInformation,
TimeSpan timeout,
bool signalAppEnd,
IEnumerable<string> extraAppArguments,
IEnumerable<(string, string)> extraEnvVariables,
CancellationToken cancellationToken = default)
{
_mainLog.WriteLine($"*** Executing '{appInformation.AppName}' on MacCatalyst ***");
var appOutputLog = _logs.Create(appInformation.BundleIdentifier + ".log", LogType.ApplicationLog.ToString(), timestamp: true);

var envVariables = new Dictionary<string, string>();
AddExtraEnvVars(envVariables, extraEnvVariables);

return await RunMacCatalystApp(appInformation, timeout, extraAppArguments ?? Enumerable.Empty<string>(), envVariables, cancellationToken);
if (signalAppEnd)
{
WatchForAppEndTag(out var appEndTag, ref appOutputLog, ref cancellationToken);
envVariables.Add(EnviromentVariables.AppEndTag, appEndTag);
}

using (appOutputLog)
{
return await RunAndWatchForAppSignal(() => RunMacCatalystApp(
appInformation,
appOutputLog,
timeout,
extraAppArguments ?? Enumerable.Empty<string>(),
envVariables,
cancellationToken));
}
}

public async Task<ProcessExecutionResult> RunApp(
Expand All @@ -74,6 +91,7 @@ public async Task<ProcessExecutionResult> RunApp(
IDevice device,
IDevice? companionDevice,
TimeSpan timeout,
bool signalAppEnd,
IEnumerable<string> extraAppArguments,
IEnumerable<(string, string)> extraEnvVariables,
CancellationToken cancellationToken = default)
Expand All @@ -84,6 +102,7 @@ public async Task<ProcessExecutionResult> RunApp(

var isSimulator = target.Platform.IsSimulator();
using var crashLogs = new Logs(_logs.Directory);
var appOutputLog = _logs.Create(appInformation.BundleIdentifier + ".log", LogType.ApplicationLog.ToString(), timestamp: true);

ICrashSnapshotReporter crashReporter = _snapshotReporterFactory.Create(
_mainLog,
Expand All @@ -93,57 +112,71 @@ public async Task<ProcessExecutionResult> RunApp(

_mainLog.WriteLine($"*** Executing '{appInformation.AppName}' on {target.AsString()} '{device.Name}' ***");

if (isSimulator)
string? appEndTag = null;
if (signalAppEnd)
{
simulator = device as ISimulatorDevice;
companionSimulator = companionDevice as ISimulatorDevice;
WatchForAppEndTag(out appEndTag, ref appOutputLog, ref cancellationToken);
}

if (simulator == null)
using (appOutputLog)
{
if (isSimulator)
{
_mainLog.WriteLine("Didn't find any suitable simulator");
throw new NoDeviceFoundException();
simulator = device as ISimulatorDevice;
companionSimulator = companionDevice as ISimulatorDevice;

if (simulator == null)
{
_mainLog.WriteLine("Didn't find any suitable simulator");
throw new NoDeviceFoundException();
}

var mlaunchArguments = GetSimulatorArguments(
appInformation,
simulator,
extraAppArguments,
extraEnvVariables,
appEndTag);

result = await RunSimulatorApp(
mlaunchArguments,
crashReporter,
simulator,
companionSimulator,
appOutputLog,
timeout,
cancellationToken);
}
else
{
var mlaunchArguments = GetDeviceArguments(
appInformation,
device,
target.Platform.IsWatchOSTarget(),
extraAppArguments,
extraEnvVariables,
appEndTag);

result = await RunDeviceApp(
mlaunchArguments,
crashReporter,
device,
appOutputLog,
extraEnvVariables,
timeout,
cancellationToken);
}

var mlaunchArguments = GetSimulatorArguments(
appInformation,
simulator,
extraAppArguments,
extraEnvVariables);

result = await RunSimulatorApp(
mlaunchArguments,
crashReporter,
simulator,
companionSimulator,
timeout,
cancellationToken);
return result;
}
else
{
var mlaunchArguments = GetDeviceArguments(
appInformation,
device,
target.Platform.IsWatchOSTarget(),
extraAppArguments,
extraEnvVariables);

result = await RunDeviceApp(
mlaunchArguments,
crashReporter,
device,
extraEnvVariables,
timeout,
cancellationToken);
}

return result;
}

private async Task<ProcessExecutionResult> RunSimulatorApp(
MlaunchArguments mlaunchArguments,
ICrashSnapshotReporter crashReporter,
ISimulatorDevice simulator,
ISimulatorDevice? companionSimulator,
ILog appOutputLog,
TimeSpan timeout,
CancellationToken cancellationToken)
{
Expand Down Expand Up @@ -182,72 +215,72 @@ private async Task<ProcessExecutionResult> RunSimulatorApp(

_mainLog.WriteLine("Starting test run");

return await _processManager.ExecuteCommandAsync(mlaunchArguments, _mainLog, timeout, cancellationToken: cancellationToken);
return await RunAndWatchForAppSignal(() => _processManager.ExecuteCommandAsync(
mlaunchArguments,
_mainLog,
appOutputLog,
appOutputLog,
timeout,
cancellationToken: cancellationToken));
}

private async Task<ProcessExecutionResult> RunDeviceApp(
MlaunchArguments mlaunchArguments,
ICrashSnapshotReporter crashReporter,
IDevice device,
ILog appOutputLog,
IEnumerable<(string, string)> extraEnvVariables,
TimeSpan timeout,
CancellationToken cancellationToken)
{
var deviceSystemLog = _logs.Create($"device-{device.Name}-{_helpers.Timestamp}.log", LogType.SystemLog.ToString());
var deviceLogCapturer = _deviceLogCapturerFactory.Create(_mainLog, deviceSystemLog, device.Name);
using var deviceSystemLog = _logs.Create($"device-{device.Name}-{_helpers.Timestamp}.log", LogType.SystemLog.ToString());
using var deviceLogCapturer = _deviceLogCapturerFactory.Create(_mainLog, deviceSystemLog, device.Name);
deviceLogCapturer.StartCapture();

try
{
await crashReporter.StartCaptureAsync();
await crashReporter.StartCaptureAsync();

_mainLog.WriteLine("Starting the app");
_mainLog.WriteLine("Starting the app");

var envVars = new Dictionary<string, string>();
AddExtraEnvVars(envVars, extraEnvVariables);
var envVars = new Dictionary<string, string>();
AddExtraEnvVars(envVars, extraEnvVariables);

return await _processManager.ExecuteCommandAsync(
mlaunchArguments,
_mainLog,
timeout,
envVars,
cancellationToken: cancellationToken);
}
finally
{
deviceLogCapturer.StopCapture();
deviceSystemLog.Dispose();
}
return await RunAndWatchForAppSignal(() => _processManager.ExecuteCommandAsync(
mlaunchArguments,
_mainLog,
appOutputLog,
appOutputLog,
timeout,
envVars,
cancellationToken: cancellationToken));
}

private MlaunchArguments GetCommonArguments(
AppBundleInformation appInformation,
private static MlaunchArguments GetCommonArguments(
IEnumerable<string> extraAppArguments,
IEnumerable<(string, string)> extraEnvVariables)
IEnumerable<(string, string)> extraEnvVariables,
string? appEndTag)
{
string appOutputLog = _logs.CreateFile($"{appInformation.BundleIdentifier}.log", LogType.ApplicationLog);
string appErrorOutputLog = _logs.CreateFile($"{appInformation.BundleIdentifier}.err.log", LogType.ApplicationLog);

var args = new MlaunchArguments
{
new SetStdoutArgument(appOutputLog),
new SetStderrArgument(appErrorOutputLog),
};
var args = new MlaunchArguments();

// Arguments passed to the iOS app bundle
args.AddRange(extraAppArguments.Select(arg => new SetAppArgumentArgument(arg)));
args.AddRange(extraEnvVariables.Select(v => new SetEnvVariableArgument(v.Item1, v.Item2)));

if (appEndTag != null)
{
args.Add(new SetEnvVariableArgument(EnviromentVariables.AppEndTag, appEndTag));
}

return args;
}

private MlaunchArguments GetSimulatorArguments(
private static MlaunchArguments GetSimulatorArguments(
AppBundleInformation appInformation,
ISimulatorDevice simulator,
IEnumerable<string> extraAppArguments,
IEnumerable<(string, string)> extraEnvVariables)
IEnumerable<(string, string)> extraEnvVariables,
string? appEndTag)
{
var args = GetCommonArguments(appInformation, extraAppArguments, extraEnvVariables);
var args = GetCommonArguments(extraAppArguments, extraEnvVariables, appEndTag);

args.Add(new SimulatorUDIDArgument(simulator.UDID));

Expand All @@ -271,14 +304,15 @@ private MlaunchArguments GetSimulatorArguments(
return args;
}

private MlaunchArguments GetDeviceArguments(
private static MlaunchArguments GetDeviceArguments(
AppBundleInformation appInformation,
IDevice device,
bool isWatchTarget,
IEnumerable<string> extraAppArguments,
IEnumerable<(string, string)> extraEnvVariables)
IEnumerable<(string, string)> extraEnvVariables,
string? appEndTag)
{
var args = GetCommonArguments(appInformation, extraAppArguments, extraEnvVariables);
var args = GetCommonArguments(extraAppArguments, extraEnvVariables, appEndTag);

args.Add(new DisableMemoryLimitsArgument());
args.Add(new DeviceNameArgument(device));
Expand Down
Loading

0 comments on commit c1f8287

Please sign in to comment.