[tests] Report errors last in RunApkTests
#1596
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Context: https://jenkins.mono-project.com/job/xamarin-android-pr-builder/2963/testReport/ A build for PR dotnet#1489 ran 1110 tests; it *should* have run over 35,000. Where are the missing tests? What's missing are the `.apk` on-device tests; BCL tests make up *most* of our expected test count. Analysis of the build log shows that the on-device tests *are* running, with test results being downloaded: …/android-toolchain/sdk/platform-tools/adb -s emulator-5570 pull "/data/data/Xamarin.Android.Bcl_Tests/files/.__override__/TestResults.xUnit.xml" "…/xamarin-android/tests/../bin/TestDebug/TestResult-Xamarin.Android.Bcl_Tests.xunit.xml" ... /data/data/Xamarin.Android.Bcl_Tests/files/.__override__/TestResults.xUnit.xml: 1 file pulled. 53.5 MB/s (3273412 bytes in 0.058s) but those results don't appear on the `testReport` URL. The problem is that the `RenameApkTestCases` target is never being executed, and the `RenameApkTestCases` is responsible for copying the `TestResult*` files where Jenkins looks for them: Done building target "ReleaseAndroidTarget" in project "RunApkTests.targets" -- FAILED.: The `RenameApkTestCases` target executes *after* `ReleaseAndroidTarget`, but because `ReleaseAndroidTarget` failed, the `RenameApkTestCases` target isn't executed. Meanwhile, `ReleaseAndroidTarget` reports an error to ensure that errors aren't overlooked; see commit 3b893cd. Allow the `RenameApkTestCases` target to execute by adding a new `ReportComponentFailures` target. The `ReleaseAndroidTarget` will no longer report errors, allowing the `RenameApkTestCases` target to execute, while if any tests *do* fail, the `ReportComponentFailures` target will generate an appropriate error. Furthermore, update all the `<MSBuild/>` invocations within `RunTests.targets` and add `ContinueOnError="ErrorAndContinue"` to all of them, to help ensure that we run all the tests we have, even if previous tests fail.
jonpryor
force-pushed
the
jonp-apk-errors-last
branch
from
0ce653a to
b1256fe
Compare
jonathanpeppers
approved these changes
Apr 24, 2018
jonpryor
added a commit
that referenced
this pull request
Apr 25, 2018
Reverts commit 4a85391 Commit 4a85391 [broke the build][0]: [0]: https://jenkins.mono-project.com/view/Xamarin.Android/job/xamarin-android/967/ xamarin-android/build-tools/scripts/TestApks.targets(223,5): error : Root element is missing ... ERROR: Step ‘Publish xUnit test result report’ aborted due to exception: org.xml.sax.SAXParseException; lineNumber: 1; columnNumber: 1; Premature end of file. at com.sun.org.apache.xerces.internal.parsers.DOMParser.parse(DOMParser.java:257) at com.sun.org.apache.xerces.internal.jaxp.DocumentBuilderImpl.parse(DocumentBuilderImpl.java:339) at org.jenkinsci.lib.dtkit.util.converter.ConversionService.convert(ConversionService.java:316) Caused: org.jenkinsci.lib.dtkit.util.converter.ConversionException: Error to convert - A file not found at org.jenkinsci.lib.dtkit.util.converter.ConversionService.convert(ConversionService.java:358) ... Finished: FAILURE Revert until we can better test things.
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Context: https://jenkins.mono-project.com/job/xamarin-android-pr-builder/2963/testReport/
A build for PR #1489 ran 1110 tests; it should have run over 35,000.
Where are the missing tests?
What's missing are the
.apkon-device tests; BCL tests make upmost of our expected test count.
Analysis of the build log shows that the on-device tests are
running, with test results being downloaded:
but those results don't appear on the
testReportURL.The problem is that the
RenameApkTestCasestarget is never beingexecuted, and the
RenameApkTestCasesis responsible for copying theTestResult*files where Jenkins looks for them:The
RenameApkTestCasestarget executes afterReleaseAndroidTarget, but becauseReleaseAndroidTargetfailed, theRenameApkTestCasestarget isn't executed.Meanwhile,
ReleaseAndroidTargetreports an error to ensure thaterrors aren't overlooked; see commit 3b893cd.
Allow the
RenameApkTestCasestarget to execute by adding a newReportComponentFailurestarget. TheReleaseAndroidTargetwill nolonger report errors, allowing the
RenameApkTestCasestarget toexecute, while if any tests do fail, the
ReportComponentFailurestarget will generate an appropriate error.