Using Mocha JSON or JEST/JUNIT XML as input, jahia-reporter is a CLI tool built with OCLIF to various perform actions following a test execution (send results to testrail, to slack, ...).
The main idea is to ingest a diverse set of test reports (Junit XML, mocha) and import those in a common data model. The data can then be manipulated/processed by various utilities to obtain a standardized, platform-agnostic output.
For example, if slack notifications are sent by Bamboo, CircleCI, GitHub Actions, ... following tests executions, all will look slightly different in their output, with different capabilities when it comes to the content of the message being sent. With Jahia-reporter, those can be standardized to one common messaging, no matter which platform the tests were executed in.
Additional data ingesters can easily be added to process test results from more sources.
If you need to design another ingester, or another command making use of the ingested data, you can find the exact specs of the model in src/global.type.ts (interfaces pre-fixed with JR).
The model is composed of the following levels:
- JRRun: A Jahia Reporter run, composed of multiple reports
- JRReport: A report is composed of multiple test suites
- JRTestsuite: A test suite is composed of multiple test cases
- JRTestcase: A test case is composed of 0 or more failures
- JRTestfailure: Failure in one of the test
The most important piece of analysis is JRTestcase which is an individual test that has the status PASS or FAIL depending of the presence of failures. From that point on, all metrics are bubbled-up all the way to JRRun.
Main commands are stored under src/commands, those commands must all make use of the data model and use test report as their primary data source.
Utilities commands are available in src/commands/utils, those are performing satellite activities (such as fetching installed modules versions).
More details can be obtained about all of the commands by using --help with the command (or by looking at the source code).
Given an existing project name, submit data to testrail. All of the missing elements (milestones, runs, cases) are automatically created if non-existing. Projects are not automatically created.
Given a Slack webhook, submit a formatted message to slack to report about test execution status.
Sample:
Test summary for: MODULE_NAME - 35 tests - 4 failures
Suite: TEST-org.jahia.test.graphql.GraphQLPublicationTest.xml - 6 tests - 4 failures
|-- shouldPublishNoSubtree(org.jahia.test.graphql.GraphQLPublicationTest) (5s) - 2 failures
|-- shouldGetErrorNotRetrieveAggregatedPublicationInfoFromLive(org.jahia.test.graphql.GraphQLPublicationTest) (3s) - 2 failures
Various parameters are available to customize the message (notification icon, ...), notify particular individuals, ...
Given a ZenCrepes webhook, sends the outcome of a test run to ZenCrepes with the objective of building a test matrix of all latest runs for all possible combinations (Augmented Search 2.1.0, tested on 2020-08-10, with Jahia 8.0.1.0 and Elasticsearch 7.8.0).
Given a Jahia host (and credentials), fetches the Jahia version (and build number) and version of all modules and save this results in a JSON file. Data model used is detailed here: src/global.type.ts
This file can then be used as input for other Jahia-Repoter first-level commands.
To add a new command, simply create the corresponding .ts file in the ./src/commands/ folder.
yarn
./bin/run yourcommand --help
If adding a utility, please add this into the utils folder
CI/CD on this repository is performed with GitHub actions, publication to NPM and building of the docker images can be done by creating a release in GitHub(with three digit versioning).
npm install -g @jahia/jahia-reporter@latest
jahia-reporter testrail --help
docker run jahia/jahia-reporter:latest jahia-reporter slack --help
Since the tool takes actual files as primary input, it is necessary to make them available to the tool during the run, this can be done by mounting the volume like this:
docker run --rm -v $PWD/junit/json2:/root/:ro jahia/jahia-reporter:latest /bin/bash -c "jahia-reporter slack /root/ WEBHOOK_URL -t json -u http://www.circleci.com -n \<@SLACK_USERNAME\> -w jahia -m augmented-search"