[Proposal] Request to add support for CI/CD pipelines for Test reporting

[nigurr]

Issue:

Currently Azure DevOps and Circle CI/CD systems provide rich test experience.
In order to get this experience/track test reporting, users has to explicitly generate test reports (currently supported format JUnit) by 3rd party plugins and publish test results.

Most of the popular open source projects aren't using 3rd party reporting plugins.
To keep local development experience and CI/CD build experience similar, scripts to build/test are written to print test results to console (std_out).
To get richer test experience in CI/CD pipeline, customers has to modify their scripts and often need to add 3rd party plugins.
This is a steep learning and test reporting is one of the important data that should be easily available for open source github projects.

Proposal for solution(s):

Generate test results file based on environment variables:
In this approach, test runners will generate test results file along with regular console output if an environment variable is set.
CI/CD pipelines can set this variable to desired test output location where they can automatically publish test results by consuming test results file generated by test runners (Mocha, Jest, Pytest, Jasmine).
This simple flag will make sure test runners are not going through any change in behavior in terms of execution or context they are running and users don't require go through any modification to their source/scripts.

Happy to listen to community feedback. Feel free to share other ideas if proposed solution is not feasible one.

[SimenB]

There is jest-junit, but you think that should be included as part of Jest? You'd probably have to tell CI where the junit.xml file is anyways, so I'm not sure how big the gain will be? It would then also be coupled to jest's (slow) release cycle.

/cc @palmerj3

[rickhanlonii]

@nigurr thanks for the suggestion - note that we don't really consider jest-community packages as third-party as they're maintained by the jest core team and contributors in the community

I don't agree that the jest-junit code should be included in jest core because most users will not use it.

Installing jest-junit and including it in the builds you want it output in (either with the config or by command line --reporters=junit) seems like a pretty frictionless way to opt-in to the reporter and probably the way we would do it even if the actual code was in core

[rickhanlonii]

Also, it would be great to update the description to match our Feature Proposal template

[nigurr]

@rickhanlonii @SimenB thanks for your inputs.
We have done some of the analysis on how many users are actually using Jest, Jest with custom reporters.
Statistics shows that only 20% of the users are using custom reporters while rest of them are using the Jest-Core which is writing the test results to std_out.
We can go in depth to understand how rest of the 80% users are using CI pipeline or not and how they are looking to report results if they are using.

We can debate on whether Jest-Junit should become the core of Jest or not?
For example, PyTest, PHPUnit and Java test eco system does support JUnit out of the box.

Motivation

Friction less user test reporting experience in CI/CD pipeline.

In general, users doesn't require report file in local dev environment.
And they take the same experience to CI/CD pipeline by writing shell script/make file to lift away all the burden.
Now it wouldn't be easy to educate customers to understand what are all features your CI/CD solution provides and how to leverage them. It's a steep learn.

Wouldn't it be great if CI/CD solution just gets all the results and all user has to do is kick off the pipeline with bare minimum configuration.

For example: If we consider Jest CI configuration,

We added extra steps and configuration to get the results pumping out. It would be much easier to lift that from user.

I really appreciate the feedback, love to be part of this conversation and making better experiences for user.

[SimenB]

How would we know where to put the resulting xml file for CI to pick it up? One thing is telling Jest to output the report (easy enough), another thing is how to get the CI service to pick it up (potentially painful, and not scalable)

[rickhanlonii]

@SimenB jest-junit supports specifying the output file via JEST_JUNIT_OUTPUT so presumably the CI system would set something like JEST_JUNIT=1 JEST_JUNIT_OUTPUT=/path/to/xml and then they could automatically pick it up if it's output - is that what you're thinking @nigurr?

If it's possible to do this with no configuration change for the user (for example without editing circle.yml to add envars) then I'm v interested in exploring

@nigurr do you expect CI systems to make a change to provide these envars by default?

[palmerj3]

I wouldn't be against adding jest-junit directly to jest. I just don't buy the argument that it's a requirement.

[SimenB]

Somewhat related: #3536

[nigurr]

@SimenB @rickhanlonii this can be driven through Environment variable.
For example JEST_OPTS : --reporter Junit, By default the test results will generated in the current working directory and this is scoped in all CI/CD pipelines and easy to search */Junit*.xml
An advanced version could be JEST_OPTS : --reporter Junit --reportDir ./results/

The environment variable JEST_OPTS can be generic so that it can be extended for other capabilities in future. This pretty similar to PYTEST_ADDOPTS for Pytest.

CI/CD systems will set the required environment variables by default, user can still override through their scripts.
This is currently being followed in all CI/CD pipelines to identify tools etc.

Since Multiple test runners across language are supporting Junit and pretty much becoming standard report, I would favor JUnit is being the default reporting schema.
If Jest introduces their own JSON/XML formatted output reporting schema, it should be ok as CI/CD systems do support multiple reporting formats.
Important step is to generate report with lease effort from user and achieve richer test results support in CI/CD pipelines.

[rickhanlonii]

@nigurr do you expect CI systems to make a change to provide these envars by default?

[nigurr]

@rickhanlonii Yes. CI/CD pipelines currently designed to drive configuration through environment variables. Pipelines love to help leveraging richer test reporting without user intervention/trying to figure out how to achieve simple test results publishing.

As part of Azure Pipelines, we love to help community/users to achieve the same.

[SimenB]

If CIs start exposing e.g. JUNIT_REPORT_LOCATION (or whatever) automatically, we could get it into e.g. https://www.npmjs.com/package/env-ci and act upon it. However, until CIs actually expose something like this, I think it's premature for Jest to do anything.

• AppVeyor wants us to use http or shell out to push it: https://www.appveyor.com/docs/running-tests/#uploading-xml-test-results (https://www.npmjs.com/package/jasmine-reporters can be used for now)
• Circle wants a junit report in reports/junit/js-test-results.xml
• Unless my googling is failing me, neither Travis nor Semaphore supports anything like it
• Shippable seems to be similar to Circle: http://docs.shippable.com/ci/nodejs-continuous-integration/#test-and-code-coverage
• TeamCity seems to require special output to stdout, and link to https://www.npmjs.com/package/jest-teamcity-reporter in their docs
• Jenkins seemingly needs a ton of config: https://medium.com/@elisegev/running-tests-and-creating-code-coverage-reports-for-react-nodejs-project-continuously-with-60312b6a2dd0
• A bunch more CIs I didn't bother to check most likely have their own ways

Hard-coding all of this into Jest seems unrealistic. Either CIs have to provide the needed configuration in the environment, or I think this is better solved on a case-by-case basis in user-land. Coupling Jest to a bunch of different CIs doesn't seem scalable. I can imagine the issues for "what about my CI of choice?" already.

---

It would be nice if they also supported the same thing for coverage reports, although that might be out of scope for this?

[nigurr]

@SimenB
Currently CircleCI, Azure DevOps, Team City has richer test experience.

• CircleCI supports publishing of test results (Junit format), however they require user to publish/copy to specific test artifacts folder.
• Azure DevOps supports in similar fashion. We are extending this to publish from well known folders automatically.
• Currently team city is not open for public projects like Github, requires team city packages to get rich results experience.

We are looking out for solution where if JEST_OPTS are provided in environment variable add the option to Jest runner automatically.
This doesn't require any npmjs.com/package/env-ci external packages.
Jest doesn't require any knowledge of running in CI pipeline or where about.
Its pipelines responsibility to leverage how to use JEST_OPTS to enhance this experience.

For example:
Azure DevOps/Circle CI might simply add JEST_OPTS = --reporter Junit --outputFolder $(testartifacts) -> the result file be used by pipelines to publish reporting.
TeamCity might simply add JEST_OPTS = --reporter teamCity --plugin somePlugin
and so on.

CI pipelines can essentially add their own specific folders to get the reporting as per their requirements.
CircleCI, Azure DevOps can get benefited immediately with this change.

Regarding coverage, yes it will be in similar lines.
For example JEST_OPTS == --reporter Junit --outputFolder $(testartifacts) --enableCoverage --coverageFolder $(coverageArtifacts)

Pipelines can now effectively publish both test results and code coverage. It can be extended to any other reporting/enabling/disabling some custom reporter/logger.

We (JEST) shouldn't be understanding CI/CD persona, runner should simply read environment variable Jest_Opts like console argument.
This design makes it extensible for 3rd party to pass extra arguments based on their requirements (please look into PyTest example, it's exactly in the same line).
Jest will remain as it's and untouched by pipelines concept.

Please let me know if I can better document it.

[SimenB]

If CIs expose a JEST_OPTS, that should be something we could hook into. Would you be able to get agreement from other CI providers to implement this?

---

We currently don't have support for defining coverageDirectory or passing options to reporters on the CLI, but we could always add that, or parse some sort of JEST_OPTS to add it. However, we should probably just support all config options as cli options (might be clunky for advanced options) anyways.

However, I'm not sure about bundling a bunch of reporters with Jest - I'm afraid that our slow release cycle might get in the way of good integrations. You could pass --reporter /some/path/to/reporter.js, but I'm not sure how we would handle differing Jest versions and compatible reporters. Maybe include some sort of reporterApiVersion?

Not sure if either of those last two paragraphs are feasible, or good ideas, but would love your thoughts on it.

[nigurr]

1. As part of Azure DevOps team, we will be able to leverage on this. The solution is open for every CI provider to leverage.
2. Coverage can be extended in similar fashion, we can have deeper conversation for it as follow up.
3. We can bring in Jest-Junit reporter and provide results out of the box. The reporting schema didn't change (it's been same for quite some year). It's one time effort and subsequently it will be minor bug fixes.
   Release cycle shouldn't become issue as long as we have tests to cover reporting. Looking from release cycle and number of issues (jest-junit), this is solid/stable product.
4. Jest can product test results in their own Json/Xml format, however it brings fragmentation in reporting. Most of the open source test reporting/CI pipeline offerings trending towards Junit.
5. Alternate option is reporter can be added through environment option and each CI pipeline can run write their own reporter to leverage to their needs. If and only if reporter and actual test code doesn't have dependency.

To conclude,

1. Adding environment option is must and will benefit any approach we take.
2. Bringing Junit into Jest core adds lot of value, unifying test reporting format across multiple languages (a step in the right direction!)
3. Reporter is an alternate approach but its beneficial if test code and reporter can be completely decoupled.

[SimenB]

As part of Azure DevOps team, we will be able to leverage on this. The solution is open for every CI provider to leverage.

Yeah, but will they? I've never had a feature request be answered by either Circle or Travis, so I'm not sure how feasible it is. And building support for something very few CIs will use might not be worth it. I might be wrong though! 🙂

Coverage can be extended in similar fashion, we can have deeper conversation for it as follow up.

👍

The reporting schema didn't change (it's been same for quite some year). It's one time effort and subsequently it will be minor bug fixes.

I meant the contract between Jest and reporters for it, not the junit format itself. Sorry if I as unclear! This is relevant as we might e.g. move to an event based architecture for reporters instead of call functions on it manually, see #6616

Jest can product test results in their own Json/Xml format, however it brings fragmentation in reporting. Most of the open source test reporting/CI pipeline offerings trending towards Junit.

We don't want that (although junit is limited, we don't want to invent the wheel here)

Bringing Junit into Jest core adds lot of value, unifying test reporting format across multiple languages (a step in the right direction!)

jest-junit is a reporter, at least in Jest's context 🙂 I think I understand what you mean though

[palmerj3]

Even if we did add JEST_OPTS environment support there is no guarantee of compatibility. We're currently discussing simplifying the configs and this may result in changes to CLI. New features are also affected by this. Azure would have to have some pretty deep knowledge of jest and know the jest version of a project before setting JEST_OPTS and I think that's a risk to user experience.

One alternative, from my perspective, would be that github detects you are using jest and comments on a PR with recommended configuration that a user/team could apply. That would increase usage and still keep configuration in user land.

[SimenB]

I'd prefer some env variable saying "test reports in this format should go to this directory, coverage in this format to this directory".

[palmerj3]

Yeah something like that makes a lot of sense. A CI standard optional config overrides.

[nigurr]

I'd prefer some env variable saying "test reports in this format should go to this directory, coverage in this format to this directory".

+1 exactly. Can we agree on this and support environment variable option to accept?

[SimenB]

TEST_REPORT_DIRECTORY & TEST_REPORT_FORMAT? And COVERAGE_REPORT_DIRECTORY & COVERAGE_REPORT_FORMAT

[nigurr]

Would suggest TEST_REPORTER and COVERAGE_REPORTER :)

[github-actions]

This issue is stale because it has been open for 1 year with no activity. Remove stale label or comment or this will be closed in 14 days.

[github-actions]

This issue was closed because it has been stalled for 7 days with no activity. Please open a new issue if the issue is still relevant, linking to this one.