From 3539269e371bd0cd9c8eeb13a386ca334623063b Mon Sep 17 00:00:00 2001 From: Adrian Clay Lake Date: Fri, 20 Dec 2024 14:19:42 +0100 Subject: [PATCH 1/4] feat: Reusable Workflow Docs (#304) Add reusable workflow docs to README.md --------- Co-authored-by: zhijie-yang --- README.md | 105 ++++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 105 insertions(+) diff --git a/README.md b/README.md index d0e5316e..317d2f09 100644 --- a/README.md +++ b/README.md @@ -5,6 +5,27 @@ *Behind every great rock is a great quarry...* +## Index +- [Before you get started](#-**Before-you-get-started**) + - [What is the OCI Factory?](#What-is-the-OCI-Factory?) + - [Why does it exist?](#Why-does-it-exist?) + - [Who is it for?](#Who-is-it-for?) + - [How to qualify as a Maintainer?](#How-to-qualify-as-a-Maintainer?) +- [How to contribute](#-How-to-contribute) + - [As a developer](#As-a-developer) + - [As a Maintainer](#As-a-Maintainer--) +- [Maintainer files](#-Maintainer-files) + - [Trigger files](#Trigger-files) + - [Image trigger file](#Image-trigger-file) + - [Documentation trigger file](#Documentation-trigger-file) + - [Other files](#Other-files) + - [Contacts](#Contacts) + - [Vulnerability Filtering](#Vulnerability-Filtering) +- [Reusable workflows](#-Reusable-workflows) + - [Build-Rock Workflow](#Build-Rock-Workflow) + - [Test-Rock Workflow](#Test-Rock-Workflow) + + ## 🍿 **Before you get started** If you are planning on contributing to this repository, you **must** first @@ -353,3 +374,87 @@ CVE-2024-0000 # private-key ``` + +## 📦 Reusable workflows + +The OCI Factory provides reusable GitHub workflows designed to support +Rock-oriented CI/CD tasks in other repositories. Currently there are two +reusable workflows available, Test-Rock and Build-Rock. As the name suggests, +these workflows are capable of building and testing rocks, and use the same +methods as the OCI Factory itself. + + +### Build-Rock Workflow + +The [Build-Rock workflow](.github/workflows/Build-Rock.yaml) +can create multi-architecture Rocks (OCI images) from a specified Rockcraft +project file (rockcraft.yaml). This project file can be located in the +repository initiating the workflow, an external repository hosted on GitHub, or +a Git repository hosted elsewhere. The resulting image is uploaded as a build +artifact in the GitHub workflow. Currently, multi-architecture builds support +`amd64` and `arm64`, depending on the availability of GitHub runners for these +architectures. Additional architectures, such as `ppc64el` and `s390x` are +supported through Launchpad build services. + +**Samples:** +- [Building an external Rock](https://github.com/canonical/rocks-toolbox/blob/main/.github/workflows/oci-factory_build_mock_rock.yaml) + - Build the `mock-rock` located in `mock_rock/1.0` +- [Build and Test EICAR Rock](https://github.com/canonical/rocks-toolbox/blob/main/.github/workflows/oci-factory_build_and_test_eicar_rock.yaml) + - Build a Rock that includes the + [EICAR test file](https://en.wikipedia.org/wiki/EICAR_test_file) and run the + Test-Rock workflow on it. The workflow is expected to fail during the + malware scan for demonstration purposes. +- [Building an external Rock](https://github.com/canonical/rocks-toolbox/blob/main/.github/workflows/oci-factory_build_external_rock.yaml) + - Build a Chiseled-Python Rock from an external repository using a specified Git commit hash. + +**Workflow Inputs:** +| Property | Required | Type | Description | +|---|---|---|---| +| `oci-archive-name` | True | str | Final filename of the rock OCI archive. | +| `build-id` | False | str | Optional string for identifying workflow jobs in GitHub UI | +| `rock-repo` | True | str | Public Git repo where to build the rock from. | +| `rock-repo-commit` | True | str | Git ref from where to build the rock from. | +| `rockfile-directory` | True | str | Directory in repository where to find the rockcraft.yaml file. | +| `arch-map` | False | JSON str | JSON string mapping target architecture to runners. | +| `lpci-fallback` | False | bool | Enable fallback to Launchpad build when runners for target arch are not available. | + +### Test-Rock Workflow + +The [Test-Rock workflow](.github/workflows/Test-Rock.yaml) +runs a series of tests on a rock or an OCI image. The image can be sourced either +from a local artifact or from an external location uploaded as an artifact. The +workflow includes the following tests, which can be enabled or disabled as +needed. + +- OCI compliance testing of images using [Umoci](https://umo.ci/). The image's + readability and layout are tested by unpacking and listing the image tags. +- Black-box testing of images performed using Docker to create a container and + attempting to run the Pebble service manager. This test applies only to + images created with Rockcraft. +- Testing image storage efficiency using [Dive](https://github.com/wagoodman/dive) +- Scanning for vulnerabilities using [Trivy](https://trivy.dev/) +- Scanning for malware using [ClamAV](https://www.clamav.net/) + +**Samples:** +- [Build and Test EICAR Rock](https://github.com/canonical/rocks-toolbox/blob/main/.github/workflows/oci-factory_build_and_test_eicar_rock.yaml) + - Build a Rock that includes the + [EICAR test file](https://en.wikipedia.org/wiki/EICAR_test_file) and run the + Test-Rock workflow on it. The workflow is expected to fail during the + malware scan for demonstration purposes. + +- [Test an External Image](https://github.com/canonical/rocks-toolbox/blob/main/.github/workflows/oci-factory_test_external_rock.yaml) + - Download and test the + [bkimminich/juice-shop](https://hub.docker.com/r/bkimminich/juice-shop) + image from Docker Hub. Note that we must skip the Black Box testing since + this is not a rock and does not include [Pebble](https://github.com/canonical/pebble). + +**Workflow Inputs:** +| Property | Required | Type | Description | +|---|---|---|---| +|`oci-archive-name`| True | str | Artifact name to download for testing. | +|`test-black-box`| False | bool | Enable rock black-box test. Enabled by default. | +|`test-oci-compliance`| False | bool | Enable Umoci OCI Image compliance test. Enabled by default. | +|`test-efficiency`| False | bool | Enable Dive image efficiency test. Enabled by default. | +|`test-vulnerabilities`| False | bool | Enable Trivy vulnerability test. Enabled by default. | +|`trivyignore-path`| False | str | Optional path to `.trivyignore` file used in vulnerability scan. | +|`test-malware`| False | bool | Enable ClamAV malware test. Enabled by default. | From 41d506ae2fd2ef01d7dffef95e443221a5bb3dd8 Mon Sep 17 00:00:00 2001 From: Adrian Clay Lake Date: Fri, 20 Dec 2024 14:57:36 +0100 Subject: [PATCH 2/4] move github vulnerability report --- .github/workflows/Test-Rock.yaml | 29 +++++++++++++++++++++++ .github/workflows/Vulnerability-Scan.yaml | 6 ----- 2 files changed, 29 insertions(+), 6 deletions(-) diff --git a/.github/workflows/Test-Rock.yaml b/.github/workflows/Test-Rock.yaml index 0a9fa1b2..f35b7cf4 100644 --- a/.github/workflows/Test-Rock.yaml +++ b/.github/workflows/Test-Rock.yaml @@ -271,6 +271,35 @@ jobs: name: ${{ steps.configure-trivy.outputs.report-name }} path: ${{ steps.configure-trivy.outputs.report-name}} + + # We have to walk through the vulnerabilities since trivy does not support outputting the results as Markdown + - name: Create markdown content + id: create-markdown + run: | + set -x + + vulnerabilities="$(jq -r -c '[ + try(.scanner.result.Results[]) + | .Target as $target + | .Vulnerabilities + | select(. != null) + | .[] + | {Target: $target, LastModifiedDate: .LastModifiedDate, VulnerabilityID: .VulnerabilityID, + PkgName: .PkgName, Severity: .Severity} + ]' < ${{ steps.configure-trivy.outputs.report-name }})" + + num_vulns=$(echo '$vulnerabilities' | jq -r 'length') + + if [[ $num_vulns -gt 0 ]]; then + echo "# Vulnerabilities found for ${{ inputs.oci-archive-name }}" >> $GITHUB_STEP_SUMMARY + title="Vulnerabilities found for ${{ inputs.oci-archive-name }}" + echo "## $title" >> $GITHUB_STEP_SUMMARY + echo "| ID | Target | Severity | Package |" >> $GITHUB_STEP_SUMMARY + echo "| -- | ----- | -------- | ------- |" >> $GITHUB_STEP_SUMMARY + echo '$vulnerabilities' | jq -r '.[] | "| \(.VulnerabilityID) | /\(.Target) | \(.Severity) | \(.PkgName) |"' >> $GITHUB_STEP_SUMMARY + fi + + test-malware: runs-on: ubuntu-22.04 name: "test-malware ${{ inputs.oci-archive-name != '' && format('| {0}', inputs.oci-archive-name) || ' '}}" diff --git a/.github/workflows/Vulnerability-Scan.yaml b/.github/workflows/Vulnerability-Scan.yaml index 2e82b05a..d81dfd95 100644 --- a/.github/workflows/Vulnerability-Scan.yaml +++ b/.github/workflows/Vulnerability-Scan.yaml @@ -213,12 +213,6 @@ jobs: echo "issue-body-file=issue.md" >> "$GITHUB_OUTPUT" fi - - name: Write to summary - if: ${{ !inputs.create-issue && steps.create-markdown.outputs.vulnerability-exists == 'true' }} - run: | - echo "# Vulnerabilities found for ${{ inputs.oci-image-name }}" >> $GITHUB_STEP_SUMMARY - cat ${{ steps.create-markdown.outputs.issue-body-file }} | tail -n +2 >> $GITHUB_STEP_SUMMARY - - id: issue-exists if: ${{ inputs.create-issue}} run: | From 7c474a19577ff05c2d80537815f58808ad079bac Mon Sep 17 00:00:00 2001 From: Adrian Clay Lake Date: Fri, 20 Dec 2024 15:50:51 +0100 Subject: [PATCH 3/4] Revert "move github vulnerability report" This reverts commit 41d506ae2fd2ef01d7dffef95e443221a5bb3dd8. --- .github/workflows/Test-Rock.yaml | 29 ----------------------- .github/workflows/Vulnerability-Scan.yaml | 6 +++++ 2 files changed, 6 insertions(+), 29 deletions(-) diff --git a/.github/workflows/Test-Rock.yaml b/.github/workflows/Test-Rock.yaml index f35b7cf4..0a9fa1b2 100644 --- a/.github/workflows/Test-Rock.yaml +++ b/.github/workflows/Test-Rock.yaml @@ -271,35 +271,6 @@ jobs: name: ${{ steps.configure-trivy.outputs.report-name }} path: ${{ steps.configure-trivy.outputs.report-name}} - - # We have to walk through the vulnerabilities since trivy does not support outputting the results as Markdown - - name: Create markdown content - id: create-markdown - run: | - set -x - - vulnerabilities="$(jq -r -c '[ - try(.scanner.result.Results[]) - | .Target as $target - | .Vulnerabilities - | select(. != null) - | .[] - | {Target: $target, LastModifiedDate: .LastModifiedDate, VulnerabilityID: .VulnerabilityID, - PkgName: .PkgName, Severity: .Severity} - ]' < ${{ steps.configure-trivy.outputs.report-name }})" - - num_vulns=$(echo '$vulnerabilities' | jq -r 'length') - - if [[ $num_vulns -gt 0 ]]; then - echo "# Vulnerabilities found for ${{ inputs.oci-archive-name }}" >> $GITHUB_STEP_SUMMARY - title="Vulnerabilities found for ${{ inputs.oci-archive-name }}" - echo "## $title" >> $GITHUB_STEP_SUMMARY - echo "| ID | Target | Severity | Package |" >> $GITHUB_STEP_SUMMARY - echo "| -- | ----- | -------- | ------- |" >> $GITHUB_STEP_SUMMARY - echo '$vulnerabilities' | jq -r '.[] | "| \(.VulnerabilityID) | /\(.Target) | \(.Severity) | \(.PkgName) |"' >> $GITHUB_STEP_SUMMARY - fi - - test-malware: runs-on: ubuntu-22.04 name: "test-malware ${{ inputs.oci-archive-name != '' && format('| {0}', inputs.oci-archive-name) || ' '}}" diff --git a/.github/workflows/Vulnerability-Scan.yaml b/.github/workflows/Vulnerability-Scan.yaml index d81dfd95..2e82b05a 100644 --- a/.github/workflows/Vulnerability-Scan.yaml +++ b/.github/workflows/Vulnerability-Scan.yaml @@ -213,6 +213,12 @@ jobs: echo "issue-body-file=issue.md" >> "$GITHUB_OUTPUT" fi + - name: Write to summary + if: ${{ !inputs.create-issue && steps.create-markdown.outputs.vulnerability-exists == 'true' }} + run: | + echo "# Vulnerabilities found for ${{ inputs.oci-image-name }}" >> $GITHUB_STEP_SUMMARY + cat ${{ steps.create-markdown.outputs.issue-body-file }} | tail -n +2 >> $GITHUB_STEP_SUMMARY + - id: issue-exists if: ${{ inputs.create-issue}} run: | From 5ee0007e14b63b95e32f68c64619336861d9a982 Mon Sep 17 00:00:00 2001 From: clay-lake Date: Fri, 20 Dec 2024 15:01:29 +0000 Subject: [PATCH 4/4] ci: automatically update oci/mock-rock/_releases.json, from https://github.com/canonical/oci-factory/actions/runs/12433493041 --- oci/mock-rock/_releases.json | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/oci/mock-rock/_releases.json b/oci/mock-rock/_releases.json index edc455e6..a831d07d 100644 --- a/oci/mock-rock/_releases.json +++ b/oci/mock-rock/_releases.json @@ -35,31 +35,31 @@ "1.1-22.04": { "end-of-life": "2030-05-01T00:00:00Z", "candidate": { - "target": "1002" + "target": "1053" }, "beta": { - "target": "1002" + "target": "1053" }, "edge": { - "target": "1002" + "target": "1053" } }, "1-22.04": { "end-of-life": "2030-05-01T00:00:00Z", "candidate": { - "target": "1002" + "target": "1053" }, "beta": { - "target": "1002" + "target": "1053" }, "edge": { - "target": "1002" + "target": "1053" } }, "1.2-22.04": { "end-of-life": "2030-05-01T00:00:00Z", "beta": { - "target": "1003" + "target": "1054" }, "edge": { "target": "1.2-22.04_beta"