-
Notifications
You must be signed in to change notification settings - Fork 537
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[tests] Add category support to the new NUnit runner, and a category to the DotNetInterfacesShouldEqualJavaInterfaces test #1963
Conversation
This test is failing rather consistently on many physical devices. See dotnet#1534 for more info.
Related: 096210c Instead of ignoring the test, should we instead consider placing a |
That does sound like a better approach, though I believe there is a bit of work also needed to ensure those category flags can be passed to the new NUnitLite test runner now being used in the runtime suite on my end. I'll try to pick this up again and make the desired changes in the coming week or two. |
The new NUnit test runner currently only allows included or excluded categories to be set in code when inheriting from Xamarin.Android.UnitTests.NUnit.NUnitTestInstrumentation. There are cases where we would like to be able to conditionally set categories at test run time however. To address this, NUnitTestInsrumentation will now attempt to process certain instrumentation extras provided from the command line. Usage: ``` adb shell am instrument -e exclude InetAccess -w (PackageName)/(InstrumentationName) ```
Xamarin.Android.UnitTests.TestInstrumentation can now retrieve all of the string extras included in the Bundle that is passed to OnCreate. The NUnit and XUnit flavored children of TestInstrumentation can now attempt to process category and other filter information from this collection.
@grendello I've attempted to address your latest set of feedback in the previous commit. I've not updated the XUnit TestInstrumentation child in this PR, as I figured this could wait until someone has a need that involves processing string extras in that runner. |
Context: #1534 The `DotNetInterfacesShouldEqualJavaInterfaces()` unit test *occasionally* fails on certain Android devices, such as: * [Oneplus One](https://gist.github.com/pjcollins/50b3036dd4a0b4a4d881a24fe49ab30d): 09-25 12:36:18.234 I/mono-stdout(26038): [FAIL] : #6.3 (dummy0 not found in Java interfaces) * [Samsung S8](https://gist.github.com/pjcollins/5782e1bc225cdf105a75491e23f257d6) 09-29 10:26:41.754 I/mono-stdout(18006): Expected: 19 09-29 10:26:41.754 I/mono-stdout(18006): But was: 5 * [Nexus 6p (API 25)](https://gist.github.com/pjcollins/062f2124e1e76561809193450f97e768) 09-29 10:26:05.053 I/mono-stdout(32432): Expected: 22 09-29 10:26:05.053 I/mono-stdout(32432): But was: 4 We do not yet understand what is causing these failures. That said, QA doesn't want to continue seeing these errors in QA builds, as reviewing "errors for which we've already filed bugs" is tedious and annoying, and makes it easier to overlook new issues. Extend the `[CategoryAttribute]` support in 096210c so that the new `TestRunner.NUnit` test runner (c4e8165) can also use the same `$(ExcludeCategories)` MSBuild property to cause tests to be skipped. Add `[Category("NetworkInterfaces")]` to the `DotNetInterfacesShouldEqualJavaInterfaces()` test. Together, these two changes will allow QA to selectively skip the `DotNetInterfacesShouldEqualJavaInterfaces()` test.
This test is failing rather consistently on many physical devices. See #1534 for more info.
I'd like to add a way to ignore this test conditionally, and we can do so with categories. However, the new NUnit test runner currently only allows included or excluded categories to be set in code when inheriting from
Xamarin.Android.UnitTests.NUnit.NUnitTestInstrumentation
. To address this,NUnitTestInstrumentation
will now attempt to process certain instrumentation extras provided from the command line.Usage:
adb shell am instrument -e include InetAccess:NetworkInterfaces -w (PackageName)/(InstrumentationName)
or
adb shell am instrument -e exclude NetworkInterfaces -w (PackageName)/(InstrumentationName)