-
Notifications
You must be signed in to change notification settings - Fork 121
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
ci: automate pinning charms in regression tests (#1215)
Switch to using pinned versions of the charm (latest main) for our charm tests in the GitHub Actions. - [x] charmcraft-pack.yaml - [x] db-charm-tests.yaml - [x] ~framework-tests.yaml~ no change needed - [x] hello-charm-tests.yaml - [x] observability-charm-tests.yaml - [x] ~publish.yml~ aggregates other workflows - [x] ~test-publish.yml~ aggregates other workflows - [x] automation to update pins - [x] detect changes - [x] create PR to update pins - [x] setup, token Also I've re-enabled mysql-k8s charm test, as that apparently got fixed upstream. Here's the configuration for the personal access token I'm using to develop the external charm "dependabot": - read/write access to the repo - `workflow` scope to push branches with changes to workflows  Co-authored-by: Tony Meyer <tony.meyer@gmail.com>
- Loading branch information
1 parent
16240bf
commit e40f78d
Showing
11 changed files
with
233 additions
and
43 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,29 @@ | ||
| --- | ||
| name: Update Charm Pins | ||
| description: Updates pinned versions of external charms we use to test our changes against to prevent regressions | ||
| author: Dima Tisnek <dimaqq@gmail.com> | ||
| branding: | ||
| icon: activity | ||
| color: orange | ||
|
|
||
| inputs: | ||
| workflows: | ||
| description: Whitespace-separated paths to the local workflow file, relative to repository root | ||
| required: true | ||
| gh-pat: | ||
| description: Personal access token to check out external repos from github | ||
| required: true | ||
|
|
||
| runs: | ||
| using: composite | ||
| steps: | ||
| - uses: actions/setup-python@v5 | ||
| with: | ||
| python-version: "3.12" | ||
|
|
||
| - run: python -m pip install -r .github/actions/update-charm-pins/requirements.txt | ||
| shell: bash | ||
| - run: python .github/actions/update-charm-pins/main.py '${{ inputs.workflows }}' | ||
| shell: bash | ||
| env: | ||
| GITHUB_TOKEN: ${{ inputs.gh-pat }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,55 @@ | ||
| # Copyright 2024 Canonical Ltd. | ||
|
|
||
| """Updates pinned versions of charms in tests.""" | ||
|
|
||
| import logging | ||
| import os | ||
| import sys | ||
|
|
||
| from httpx import Client | ||
| from ruamel.yaml import YAML | ||
|
|
||
| yaml = YAML(typ="rt") | ||
| yaml.indent(mapping=2, sequence=4, offset=2) | ||
|
|
||
| github = Client( | ||
| base_url="https://api.github.com/repos", | ||
| headers={ | ||
| "Authorization": f'token {os.getenv("GITHUB_TOKEN")}', | ||
| "Accept": "application/vnd.github.v3+json", | ||
| }, | ||
| ) | ||
|
|
||
|
|
||
| def update_charm_pins(workflow): | ||
| """Update pinned versions of charms in the given GitHub Actions workflow.""" | ||
| with open(workflow) as file: | ||
| doc = yaml.load(file) | ||
|
|
||
| # Assume the workflow has a single job or the first job is parameterized with charm repos | ||
| job_name = next(iter(doc["jobs"])) | ||
|
|
||
| for idx, item in enumerate(doc["jobs"][job_name]["strategy"]["matrix"]["include"]): | ||
| charm_repo = item["charm-repo"] | ||
| commit = github.get(f"{charm_repo}/commits").raise_for_status().json()[0] | ||
| data = github.get(f"{charm_repo}/tags").raise_for_status().json() | ||
| comment = " ".join( | ||
| [tag["name"] for tag in data if tag["commit"]["sha"] == commit["sha"]] | ||
| + [commit["commit"]["committer"]["date"]] | ||
| ) | ||
|
|
||
| # A YAML node, as opposed to a plain value, can be updated in place to tweak comments | ||
| node = doc.mlget( | ||
| ["jobs", job_name, "strategy", "matrix", "include", idx], list_ok=True | ||
| ) | ||
| node["commit"] = commit["sha"] | ||
| node.yaml_add_eol_comment(comment, key="commit") | ||
|
|
||
| with open(workflow, "w") as file: | ||
| yaml.dump(doc, file) | ||
|
|
||
|
|
||
| if __name__ == "__main__": | ||
| logging.basicConfig(level="INFO") | ||
| for workflow in " ".join(sys.argv[1:]).split(): | ||
| update_charm_pins(workflow) |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,27 @@ | ||
| # Update Charm Pins | ||
|
|
||
| ## GitHub Actions Usage | ||
|
|
||
| Inputs: | ||
|
|
||
| - `workflows`: space or newline-separated list of workflow YAML files relative to this repository root | ||
| - `gh-pat`: personal access token to query external repositories hosted at GitHub | ||
|
|
||
| This action will update the `workflows` in the current checkout. It is the responsibility of the caller | ||
| to do something with these changes. | ||
|
|
||
| ## Local Usage | ||
|
|
||
| ```command | ||
| # set up a venv and install the deps | ||
| pip install -r requirements.txt | ||
|
|
||
| # set the GITHUB_TOKEN env var with a personal access token | ||
| export GITHUB_TOKEN=ghp_0123456789 | ||
|
|
||
| # run the script | ||
| python main.py path-to/.github/workflows/one.yaml path-to/.github/workflows/another.yaml | ||
|
|
||
| # check the modifications in the current branch | ||
| git diff | ||
| ``` |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,2 @@ | ||
| ruamel.yaml==0.18.6 | ||
| httpx==0.27.0 |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,53 @@ | ||
| --- | ||
| name: Update Charm Pins | ||
|
|
||
| on: | ||
| # NOTE: to avoid infinite loop, exclude the branch created by this workflow if triggering on push or pull_request | ||
| workflow_dispatch: | ||
| schedule: | ||
| - cron: '0 18 * * SUN' # Sunday 6pm UTC, before international day starts | ||
|
|
||
| jobs: | ||
| update-pins: | ||
| runs-on: ubuntu-latest | ||
| steps: | ||
| - uses: actions/checkout@v4 | ||
| with: | ||
| token: ${{ secrets.UPDATE_CHARM_PINS_ACCESS_TOKEN }} | ||
|
|
||
| - uses: ./.github/actions/update-charm-pins | ||
| with: | ||
| # Whitespace (null) separated string, as workflow inputs are always plain values | ||
| workflows: |- | ||
| .github/workflows/db-charm-tests.yaml | ||
| .github/workflows/hello-charm-tests.yaml | ||
| .github/workflows/charmcraft-pack.yaml | ||
| .github/workflows/observability-charm-tests.yaml | ||
| gh-pat: ${{ secrets.UPDATE_CHARM_PINS_ACCESS_TOKEN }} | ||
|
|
||
| - run: | | ||
| # Force-push pin changes to the branch | ||
| echo "New changes in charm pins" | ||
| git --no-pager diff | ||
| git config --global user.name "github-actions" | ||
| git config --global user.email "github-actions@github.com" | ||
| git switch -C auto-update-external-charm-pins | ||
| git commit --allow-empty -am "chore: update charm pins" | ||
| echo "Total changes in charm pins" | ||
| git --no-pager diff main HEAD | ||
| git push -f --set-upstream origin auto-update-external-charm-pins | ||
| - run: | | ||
| # Ensure a PR if there are changes, no PR otherwise | ||
| PR=$(gh pr list --state open --head auto-update-external-charm-pins --json number -q '.[0].number') | ||
| CHANGES=$(git --no-pager diff --stat main HEAD) | ||
| echo "Existing PR? $PR" | ||
| echo "Changes? $CHANGES" | ||
| if [[ -n "$PR" && -z "$CHANGES" ]]; then | ||
| echo "Closing #$PR as stale" | ||
| gh pr close -c stale "$PR"; | ||
| elif [[ -z "$PR" && -n "$CHANGES" ]]; then | ||
| echo "Opening new PR" | ||
| gh pr create --base main --head auto-update-external-charm-pins --title "chore: update charm pins" --body "This is an automated PR to update pins of the external repositories that the operator framework is tested against"; | ||
| fi | ||
| env: | ||
| GITHUB_TOKEN: ${{ secrets.UPDATE_CHARM_PINS_ACCESS_TOKEN }} |
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters