Azure DevOps Pipelines invalid JUnit XML report format

[dadair-ca]

Description

I am trying to run kaocha with the kaocha-junit-xml plugin to output test results in the new Azure DevOps build pipelines, to test if that's a platform my company wants to move towards. The pipeline task Publish Test Results is having trouble parsing the JUnit XML produced by the kaocha-junit-xml. I am unsure if this is a Pipelines issue, a Publish Test Results task issue, or a kaocha-junit-xml issue. I am not sure if there is a JUnit schema validator somewhere to check against?

I am on the following dependency versions:

{:deps {}
 :aliases
 {:test {:extra-deps {lambdaisland/kaocha {:mvn/version "0.0-266"}
                      lambdaisland/kaocha-junit-xml {:mvn/version "0.0-27"}
                      lambdaisland/kaocha-cloverage {:mvn/version "0.0-6"}}}}}

Build output

2018-11-19T23:11:50.4796404Z ##[debug]Reading test results from file '/home/agent/AzureAgent/_work/2/s/junit.xml'
2018-11-19T23:11:50.4937649Z Only single test suite found, parsing its information
2018-11-19T23:11:50.5004824Z ##[warning]Invalid results file. Make sure the result format of the file '/home/agent/AzureAgent/_work/2/s/junit.xml' matches 'JUnit' test results format.

JUnit File

<?xml version='1.0' encoding='UTF-8'?>
<testuites skipped="0" errors="0" failures="0" tests="1">
<testsuite name="unit" id="0" hostname="localhost" skipped="0" errors="0" failures="0" tests="1">
<testcase name="helloworld.core-test/a-test" classname="helloworld.core-test" assertions="1">
</testcase>
</testsuite>
</testuites>

helloworld.zip

[plexus]

Hi @adairdavid , I've changed the output to be conform with this XML Schema which is linked from the Azure docs. This has been released as lambdaisland/kaocha-junit-xml {:mvn/version "0.0-31"}. Please try it out!

This is quite a big change, @ggiraldez could you also try it out to see if it works for you?

Thanks!

[ggiraldez]

Works for me on CircleCI.

I did notice the system-out and system-err tags are applied to the test suites instead of each test case, and thus all output is concatenated. CircleCI does not show that content anywhere, so it makes no difference to me. But I think it'd be better to have it per-testcase instead. This contradicts with the XML schema though.

[plexus]

I agree it would be useful to have it at the testcase, but that would invalidate the schema. I originally used this as a reference (waybackmachine link because apparently in the past week it stopped showing the example file). This has several things like system-out / system-err at the test-case as well as testsuite level, and skipped/assertion counts, which are all not present in the schema Azure uses.

I'm gonna leave it as is for now. If it turns out certain services aren't happy with what we generate now we can consider outputting multiple formats.

@adairdavid I'm assuming this fixes your issues. If not please reopen. Thanks!