ciclops

👁️

ciclops (CI-clops) is a GitHub Action that summarizes the result of a Test Suite executed in a series of "strategy matrix" branches, and helps orient investigations when tests fail.

The legends tell of a mythical one-eyed creature that was cursed by the gods to watch over Continuous Integration pipelines for all eternity.

Inputs

• artifact_directory: Required directory holding the JSON artifacts representing tests.

• output_file: Optional the file where the markdown report will be written. If omitted, the report will be written to the GITHUB_STEP_SUMMARY environment variable used by GitHub to display job summaries.

Outputs

Up to three outputs might be produced:

• thermometer: this will contain stand-alone text with a color-coded list of test metrics that can serve as an overview of the state of the test suite on CI/CD. This is generated on every execution of Ciclops.

• alerts: this will contain stand-alone text with systematic failures detected by CIclops. It is meant to enable further steps in the calling workflow, for example sending to Slack for monitoring of the CI/CD health.

• Overflow: this will contain the name of a file with the full Summary. This file will be created if the summary exceeded the limit of 1024K imposed by GitHub for job summaries. A shorter summary will be output, and the filename indicated by Overflow may be used by the calling workflow for Archival.

Usage

To use ciclops, your Test step(s) in your CI/CD pipeline should be producing JSON artifacts with the results of each test executed. You can find example JSON artifacts in the example_artifacts directory.

NOTE: these examples show the expected schema of the JSON artifacts. The field names are generic and should serve you. Other fields in JSON objects will be ignored.

If your CI/CD pipeline runs tests in several strategy matrix branches, you should ensure the JSON artifacts are uploaded (e.g. via the GitHub actions/upload-artifact action.) You should add the summary creation step to fire once all branches have finished, then download all the JSON artifacts created in the various branches, and gather them into one directory.

With those prerequisites, you can trigger ciclops with the artifact_directory argument set to the folder containing the JSON artifacts, and the output_file set to write a markdown report. Then you can print that report to the $GITHUB_STEP_SUMMARY environment variable provided by GitHub.

For example:

    …
    …
    steps:
      - uses: actions/checkout@v3

      - name: Create a directory for the artifacts
        run: mkdir test-artifacts

      - name: Download all artifacts to the directory
        uses: actions/download-artifact@v3
        with:
          path: test-artifacts

      - name: Flatten all artifacts onto directory
        # The download-artifact action, since we did not give it a name,
        # downloads all artifacts and creates a new folder for each.
        # In this step we bring all the JSONs to a single folder
        run: |
            mkdir test-artifacts/data
            mv test-artifacts/*/*.json test-artifacts/data

      - name: Compute the E2E test summary
        uses: cloudnative-pg/ciclops@main
        with:
          artifact_directory: test-artifacts/data

Advanced Usage

There are two advanced cases we want to call attention to:

1. Summary overflow.
   The GITHUB_STEP_SUMMARY variable set to receive the CIclops summary will overflow if the summary is bigger than 1024K. To work around this, in case of overflow CIclops will create a short version of the summary and overwrite GITHUB_STEP_SUMMARY, and in addition it will write the full summary to a file that a calling workflow can Archive. The name of the full file is sent a GitHub action output variable called Overflow.

2. Monitoring with chatops
   Ciclops will generate a "thermometer" on every execution, offering a color-coded overview of the test health. This thermometer is included in the GitHub summary, and in addition, is exported as an output in plain text, which can be sent via chatops. In addition, Ciclops will create a series of alerts when systematic failures are detected. By "systematic", we mean cases such as:

   • all test combinations have failed
   • all combinations fail for a given test
   • all tests fail for a given version of Postgres

The alerts are included in the summary generated by CIclops. In addition, CIclops will send the alerts alone as a GitHub action output variable called alerts. The alerts can then be sent via Slack message to alert DevOps teams.

The following snippet shows how to use these features:

    …
    …
    steps:
      …
      …
      - name: Generate Test Summary
        id: generate-summary
        uses: cloudnative-pg/ciclops@main
        with:
          artifact_directory: test-artifacts/data

      - name: If there is an overflow summary, archive it
        if: ${{steps.generate-summary.outputs.Overflow}}
        uses: actions/upload-artifact@v3
        with:
          name: ${{steps.generate-summary.outputs.Overflow}}
          path: ${{steps.generate-summary.outputs.Overflow}}
          retention-days: 7

      - name: Get a slack message with the Ciclops thermometer
        uses: rtCamp/action-slack-notify@v2
        env:
          SLACK_USERNAME: cnpg-bot
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
          SLACK_MESSAGE: ${{steps.generate-summary.outputs.thermometer}}

      - name: If there are alerts, send them over Slack
        if: ${{steps.generate-summary.outputs.alerts}}
        uses: rtCamp/action-slack-notify@v2
        env:
          SLACK_USERNAME: cnpg-bot
          SLACK_WEBHOOK: ${{ secrets.SLACK_WEBHOOK }}
          SLACK_MESSAGE: ${{steps.generate-summary.outputs.alerts}}
          …

Origin

At EDB, working on a series of Kubernetes operators for PostgreSQL, we have an extensive test suite that is executed for a variety of combinations of PostgreSQL and Kubernetes versions, using GitHub's strategy matrix construct.

When there are failures in a given run of our CI/CD pipeline, it becomes difficult to make sense of things. With so many tests executed in so many matrix branches, clicking into each matrix branch and drilling in to find the failing tests quickly becomes painful. Systemic issues often escape scrutiny, buried in data. Information is lost like … tears in rain.

ciclops adds a job summary to the GitHub Actions output of a CI/CD pipeline. It buckets tests according to several criteria, doing the grunt work of figuring out if there was a pattern to test failures. It also displays a table of test durations, sorted by slowest.

Contributing

Please read the code of conduct and the guidelines to contribute to the project.

Disclaimer

ciclops is open source software and comes "as is". Please carefully read the license before you use this software, in particular the "Disclaimer of Warranty" and "Limitation of Liability" items.

Copyright

ciclops is distributed under Apache License 2.0.

Copyright (C) The CloudNativePG Contributors.