Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[tests] Add category support to the new NUnit runner, and a category to the DotNetInterfacesShouldEqualJavaInterfaces test #1963

Merged
merged 6 commits into from
Jul 26, 2018

Conversation

pjcollins
Copy link
Member

@pjcollins pjcollins commented Jul 13, 2018

This test is failing rather consistently on many physical devices. See #1534 for more info.

I'd like to add a way to ignore this test conditionally, and we can do so with categories. However, the new NUnit test runner currently only allows included or excluded categories to be set in code when inheriting from Xamarin.Android.UnitTests.NUnit.NUnitTestInstrumentation. To address this, NUnitTestInstrumentation will now attempt to process certain instrumentation extras provided from the command line.

Usage:

adb shell am instrument -e include InetAccess:NetworkInterfaces -w (PackageName)/(InstrumentationName)

or

adb shell am instrument -e exclude NetworkInterfaces -w (PackageName)/(InstrumentationName)

This test is failing rather consistently on many physical devices. See
dotnet#1534 for more info.
@pjcollins pjcollins requested a review from grendello July 13, 2018 19:31
@pjcollins pjcollins requested a review from jonpryor as a code owner July 13, 2018 19:31
@jonpryor
Copy link
Member

Related: 096210c

Instead of ignoring the test, should we instead consider placing a [Category] on the test so that the test can be excluded within whatever test environment is encountering these failures?

@pjcollins
Copy link
Member Author

That does sound like a better approach, though I believe there is a bit of work also needed to ensure those category flags can be passed to the new NUnitLite test runner now being used in the runtime suite on my end. I'll try to pick this up again and make the desired changes in the coming week or two.

The new NUnit test runner currently only allows included or excluded
categories to be set in code when inheriting from
Xamarin.Android.UnitTests.NUnit.NUnitTestInstrumentation. There are
cases where we would like to be able to conditionally set categories at
test run time however. To address this, NUnitTestInsrumentation will now
attempt to process certain instrumentation extras provided from the
command line.

Usage:
```
adb shell am instrument -e exclude InetAccess -w (PackageName)/(InstrumentationName)
```
@pjcollins pjcollins changed the title [Mono.Android-Tests] Ignore DotNetJavaNetworkIfaces test [tests] Add category support to the new NUnit runner, and a category to the DotNetInterfacesShouldEqualJavaInterfaces test Jul 24, 2018
Xamarin.Android.UnitTests.TestInstrumentation can now retrieve all
of the string extras included in the Bundle that is passed to OnCreate.
The NUnit and XUnit flavored children of TestInstrumentation can now
attempt to process category and other filter information from this collection.
@pjcollins
Copy link
Member Author

@grendello I've attempted to address your latest set of feedback in the previous commit. I've not updated the XUnit TestInstrumentation child in this PR, as I figured this could wait until someone has a need that involves processing string extras in that runner.

@jonpryor jonpryor merged commit f81225d into dotnet:master Jul 26, 2018
@pjcollins pjcollins deleted the skip-ifaces branch July 26, 2018 19:53
jonpryor pushed a commit that referenced this pull request Aug 3, 2018
Context: #1534

The `DotNetInterfacesShouldEqualJavaInterfaces()` unit test
*occasionally* fails on certain Android devices, such as:

  * [Oneplus One](https://gist.github.com/pjcollins/50b3036dd4a0b4a4d881a24fe49ab30d):

        09-25 12:36:18.234 I/mono-stdout(26038): [FAIL] : #6.3 (dummy0 not found in Java interfaces)

  * [Samsung S8](https://gist.github.com/pjcollins/5782e1bc225cdf105a75491e23f257d6)

        09-29 10:26:41.754 I/mono-stdout(18006): Expected: 19
        09-29 10:26:41.754 I/mono-stdout(18006): But was: 5

  * [Nexus 6p (API 25)](https://gist.github.com/pjcollins/062f2124e1e76561809193450f97e768)

        09-29 10:26:05.053 I/mono-stdout(32432): Expected: 22
        09-29 10:26:05.053 I/mono-stdout(32432): But was: 4

We do not yet understand what is causing these failures.

That said, QA doesn't want to continue seeing these errors in QA
builds, as reviewing "errors for which we've already filed bugs" is
tedious and annoying, and makes it easier to overlook new issues.

Extend the `[CategoryAttribute]` support in 096210c so that the new
`TestRunner.NUnit` test runner (c4e8165) can also use the same
`$(ExcludeCategories)` MSBuild property to cause tests to be skipped.

Add `[Category("NetworkInterfaces")]` to the
`DotNetInterfacesShouldEqualJavaInterfaces()` test.

Together, these two changes will allow QA to selectively skip the
`DotNetInterfacesShouldEqualJavaInterfaces()` test.
@github-actions github-actions bot locked and limited conversation to collaborators Feb 3, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants