Skipped tests are not reported as skipped in xUnit output #7383
Comments
|
I wish we had a proper JUnit XML schema to follow here. I think this is as close as we have to a spec: https://github.com/junit-team/junit5/blob/main/junit-platform-reporting/src/main/java/org/junit/platform/reporting/legacy/xml/XmlReportWriter.java It does have |
|
XCTest will need to be taught to report a skipped test in a way Swift Package Manager can detect; that may ultimately need to wait until we teach XCTest how to produce JSON output like we do for Swift Testing, at which point a common implementation is possible. |
This PR adds skipped test reporting to our JUnit XML output. For example, given this test: ```swift @test(.disabled("Because I said so")) func f() {} ``` The XML output would be, approximately: ```xml <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="TestResults" errors="0" tests="1" failures="0" skipped="1" time="12345.0"> <testcase classname="MyTests" name="f()" time="12344.0"> <skipped>Because I said so</skipped> </testcase> </testsuite> </testsuites> ``` See also swiftlang/swift-package-manager#7383 which asks for this for XCTest.
This PR adds skipped test reporting to our JUnit XML output. For example, given this test: ```swift @test(.disabled("Because I said so")) func f() {} ``` The XML output would be, approximately: ```xml <?xml version="1.0" encoding="UTF-8"?> <testsuites> <testsuite name="TestResults" errors="0" tests="1" failures="0" skipped="1" time="12345.0"> <testcase classname="MyTests" name="f()" time="12344.0"> <skipped>Because I said so</skipped> </testcase> </testsuite> </testsuites> ``` See also swiftlang/swift-package-manager#7383 which asks for this for XCTest. ### Checklist: - [x] Code and documentation should follow the style of the [Style Guide](https://github.com/apple/swift-testing/blob/main/Documentation/StyleGuide.md). - [x] If public symbols are renamed or modified, DocC references should be updated.
|
@plemarquand I don't know if you are following this |
|
Currently, the XCTest implementation of xUnit (JUnit) XML output is based solely on the exit status of the process hosting each test. Additional information such as the cause of a failure is not available. Resolving this problem will require an overall refactoring of how XCTest produces its XML output. That work is tracked by #4752, so I'm going to mark this issue as a duplicate of that one. |
|
Duplicate of #4752 |
Is it reproducible with SwiftPM command-line tools:
swift build,swift test,swift packageetc?Description
The xUnit file output by
swift test --parallel --xunit-output <file>does not report when a test is skippedExpected behavior
xUnit file has an xml node that indicates the test was skipped
Actual behavior
Instead it reports that the test ran as if it passed
Steps to reproduce
try XCTSkipIf(true)swift test --parallel --xunit-output test.xmlThere is no indication in the file that test was skipped
Swift Package Manager version/commit hash
5.9.2
Swift & OS version (output of
swift --version ; uname -a)swift-driver version: 1.87.3 Apple Swift version 5.9.2 (swiftlang-5.9.2.2.56 clang-1500.1.0.2.5)
Target: arm64-apple-macosx14.0
Darwin Adams-MBP-M1-Max.local 23.2.0 Darwin Kernel Version 23.2.0: Wed Nov 15 21:53:18 PST 2023; root:xnu-10002.61.3~2/RELEASE_ARM64_T6000 arm64
The text was updated successfully, but these errors were encountered: