JUnit XML Reporting

[s9tpepper]

As a person that likes to use continuous integration tools
I would like to be able to set a -junit flag to output JUnit XML reports
So that I can have integrated CI tool reporting from cucumber.js

Ideally, it would be great to be able to do this:

cucumber.js /path/to/features -r /path/to/step_defs -junit /path/to/cucumberjs/reports

Then in /path/to/cucumberjs/reports I'd get JUnit XML reports, one for each test that is run.

I'm currently using the JUnit and the HTML reporting that is exported from Behat: http://docs.behat.org/guides/6.cli.html#format-options

[jbpros]

Thanks for filing the issue following our little chat on the Twitters, Omar.

It's interesting you posted this feature request to cucumber-tck because it could (partly) apply to all Cucumber implementations.

However, I'm not sure we want to define the same formatters for each and every Cuke.

@aslakhellesoy, @mattwynne thoughts on this?

[aslakhellesoy]

Both cucumber-jvm and cucumber-ruby have junit formatters.

Cucumber-Ruby: --format junit --out somedir
Cucumber-JVM: --format junit:somedir

I don't like the --junit option - it doesn't scale for more formatters. I prefer the way Cucumber-JVM declares output over how Cucumber-Ruby does it.

[jbpros]

@aslakhellesoy thanks -- are you saying the junit formatter could be speced in -tck? What about the other common formatters (JSON, progress, ...)?

[aslakhellesoy]

I haven't been using the TCK that much in Cucumber-JVM lately. It's fairly time consuming to tweak it to work with various implementations, and the native tests/features give me good enough confidence.

That said, JUnit XML reports are quite gnarly, both in Cucumber-Ruby and Cucumber-JVM. They don't consistently map results onto the XML schema. And there are many bugs.

What we have that is more consistent is the JSON output. I wonder if it would make more sense to write a single cucumber-json-to-junit-xml command line tool (in any language - maybe ruby) and just delete all of the junit formatters everywhere.

Then people can just run that command line tool after running Cucumber, and the JUnit reports shjould be consistent across implementations. Sort of like cucumber-html.

[s9tpepper]

I actually meant to log this to cucumber-js, sorry about that.

I do agree about the -junit flag, I like how Behat does it:

behat -format pretty,junit,html -out ,/path/to/junit/logs, /path/to/html/reports

The path for pretty print can just be left blank to print to console as usual.

I was basically hoping to find a way to integrate reports with Jenkins.

[jbpros]

Moved to cucumber-js's repo.