[Feature] Automate OpenSearch Distribution status badging

[peternied]

Problem

The OpenSearch Distribution has many sub-projects, while getting centralized metrics is important for driving releases, individual project metrics should be easy to see as well. As the project changes over time, focusing on new release versions or surfacing new metrics of importance requires making updates in N places throughout the related repositories - this will making it hard to stay up to date and manual verification of pull requests is a large amount manual effort.

Requirements

• There should be a visible part of each project to display metrics
• Metrics should be displayed in a consistent manner between projects
• Displayed metrics should be actionable, either to discover more information or directly drive action
• Which metrics that are displayed is centrally controlled
• When a metric is updated all project should be updated without human intervention
• Projects should not require any maintenance to support this system after its created
• Projects are able to surface metrics in addition to any project-wide metrics

Proposal - Automatically Badge and Update README.md

Security README.md with Badges

See https://github.com/opensearch-project/security#readme

Badges are often used in GitHub to surface metrics - https://shields.io/ was build to create these badges and has many different features and customizations. By using these badges we can create visually distinctive images and link to the sources associated with them.

README.md files surfacing the most important information to visitors of the project. adding badges onto these pages lets future users and contributors see what the project values.

A GitHub action that accepts content and updates a README.md can ensure consistency. After building a common template for a GitHub action, rolling that action to each project, there is an automated mechanism to deploy updates to the badges. OpenSearch widely uses GitHub actions for backporting PRs, detecting release manifest changes, and incrementing build versions for a new release.

Invoking all project level GitHub Actions can be triggered by a GitHub Action workflow inside of OpenSearch-build that creates the badges and supplies them as a parameter to the README.md updating workflow. This will be automated so that whenever a new release is started it automatically triggers the workflow, or when the list of badges is updated.

Milestones

README.md Update Workflow

Create a GitHub Action workflow that inserts content between html comments in markdown files. Using this workflow to effectively automate a pull request this one for OpenSearch Security [1]. This workflow can immediately push an update if the file was modified, no CI check is needed.

Badge Generation workflow

Badges are project specific, using a badging image provider and accept a project to generate a set of badges that can be feed into the update workflow. This generation process should be able to accept updates to include more badges, or altered versions.

Central badge template and workflow

There needs to be a central workflow that knows each project workflow so they can be triggered alongside the template for all badges seen in the readme.

Trigger central workflow on template updates

Add triggers for when the templates are updated to automatically deploy changes to all README.mds

Handle version updates automatically

As released of OpenSearch Distribution continue instead of having the version numbers controlled by manually updating a template file, move this process into an automated workflow that can find the most recent versions and include them.

Prototypes

Badge Creation
OpenSearch-Build-Badger [2] was created to accept a single project and generate the badges for markdown as a GitHub Action. This was used to generate the OpenSearch Security and OpenSearch README.md badges manually [1] [3]

Supported Version Detection
Knowing which versions are in progress for a release can mostly be determined in the OpenSearch-build repro, attempted to write a script to capture this state, but it needs some additional shell-fu before it can be used [4]

[1] opensearch-project/security#2167
[2] https://github.com/peternied/opensearch-build-badger
[3] opensearch-project/OpenSearch#4843
[4] opensearch-project/opensearch-build@51128d1

Additional Context

• [Operations] How to better visualize OpenSearch-Project status? #65
• POC: Concept/Solution to Automate the version increment to next development iteration. opensearch-build#2215 (comment)

[dblock]

Love it. Keeping badges green would be a visible way to track things and a separate badges dashboard could easily be built in opensearch-plugins or project-meta for example.

[ps48]

Thanks @peternied for creating initiating this proposal. Last year, we had done something similar for our plugins (Observability, Reports, SQL and Dashboards-Visualizations).

One important thing the badges were missing last year was seeing if distribution workflow from jenkins fails. Fortunately, @prudhvigodithi added auto-cut tickets for failing build we a label that we can track for each plugin Example: https://github.com/opensearch-project/job-scheduler/issues?q=is%3Aopen+is%3Aissue+label%3Aautocut

Also, let's try adding a table format as mentioned below(This needs more real-estate but is worth it for the readability). Adding below some links and table setup to our existing badges.

Badge Creation

• Create Custom Badges: https://shields.io/
• Create Workflow Badges: https://docs.github.com/en/actions/monitoring-and-troubleshooting-workflows/adding-a-workflow-status-badge
• Create Code Coverage Badges: https://docs.codecov.com/docs/status-badges
• Templates of commonly used Badges: https://github.com/Naereen/badges
• More details into awesome badges: https://github.com/badges/awesome-badges#awesome-badges

Badge types currently used in our repositories

1. Build
   1. Build plugin packages
2. Tests
   1. UTs
   2. ITs
   3. Code Coverage
   4. BWC Tests
   5. Distribution build tests
3. Issues
   1. Good first issue
   2. Feature
   3. Enhancement
   4. Bug
   5. Untriaged
   6. No Label

Observability Badges

https://github.com/opensearch-project/observability#code-summary

[dblock]

[Catch All Triage - 1, 2, 3, 4, 5]