Output check/threshold results in a machine-readable unit test format to publish test results in CI

[marlukjanska]

We're setting up k6 in our Azure DevOps CI pipeline which allows publishing predefined unit test formats results: https://docs.microsoft.com/en-us/azure/devops/pipelines/tasks/test/publish-test-results?view=azure-devops&tabs=yaml

So I was thinking of adding on teardown function to write out those test results in JUnit result format like here.

Having said that - was wondering if that would be the best approach or maybe something like this is already available?

Thanks!

[mstoykov]

As mentioned in #666 (:smiling_imp:) , k6 currently doesn't have functionality to write to file. And there are still no plans to add such for the same portability, security and scalability reasons.
Additionally as mentioned in #351 there is no way to get if the thresholds passed of didn't from inside the test or any data from the metrics as well.

You can possibly use the json output or the csv output after we merge #1067 , and than do some processing on the tests for example with jq and than have it output and transform it to junit results, but this in itself seems like a lot of work and depending on how big your tests are you might need to wait for the #1114 in order to at least have gzip compression when writing the json so you don't run out of disk space.

IMO the best idea is for us to return the json report ( #355 ) and add thresholds there. There have been other people asking for this and it has been climbing in the priority list. I am not so certain that we will directly support JUnit as ... most of the output won't be compatible but maybe the checks and thresholds are a good candidate for a separate JUnit report that is just those. Looking at the supported formats all of them are popular but JUnit might still be popular-er and be the best choice for us.

[na--]

Yeah, it makes sense to me to have the results from any thresholds in the future end-of-test machine-readable report that k6 will produce. We still haven't discussed the exact format of that report, and even though I doubt we'll be able to use a standardized format for this, some research won't hurt...

In any case, a custom k6 json format for all k6 data and a separate standardized format (JUnit, or more likely - XUnit?) for pass/fail criteria (i.e. thresholds and maybe checks?), so that can automatically be parsed by CI systems also makes a lot of sense to me...

[simskij]

Some things I think we should consider:

• JUnit XML is the most commonly supported output format for CI test report visualisation, so I'd be all for adding it as an output format.
• There is a community project already, but having it natively in k6 would definitely make a lot of sense, and likely be less error-prone.
• Having the option to decide whether to report failed checks and thresholds as warnings or errors would be really valuable, as it would allow the user to decide whether a failed k6 test should break the build or not. With that said, I think both should be available as possible build-breakers.

[na--]

I probably should create a separate issue for that, but it's likely that all sorts of "k6 generates some sort of summary at the end of a test run" will be implemented with templates. The idea was originally discussed in #1319 (comment), but it's generally applicable.

Instead of k6 having to wrangle the same pieces of data in multiple different formats (end-of-test text summary, end-of-test JSON summary, JUnit XML, text executive report, etc.), we should just support templates. Then, we can support the common "report" cases via some built-in templates, but also allow users to expose the k6 results however they want to, with custom templates.

[na--]

Now that #1768 is merged, this should be easy to do. @mstoykov already (#1768 (comment)) made a prototype: https://gist.github.com/MStoykov/5e052293cc1bbdd11284a2cc5a90a194

We'll close the issue when we polish that prototype and upload it to https://jslib.k6.io