feat: Support policies in folders for reusable workflows #138

viccuad · 2024-12-06T13:42:30Z

Description

Fix kubewarden/kubewarden-controller#806

Recommend review by commit.

See also working monorepo repo at https://github.com/viccuad/rego-policies.

Test

For monorepo policies:

Successfully failing release with tag ControllerContainerBlockSSHPort/v0.0.3
because artifacthub-pkg.yml hasn't been updated:
https://github.com/viccuad/rego-policies/actions/runs/12352787565/job/34470569919
Successful run and release with tag ControllerContainerBlockSSHPort/v0.0.4:
https://github.com/viccuad/rego-policies/actions/runs/12352849151

For Normal policies:

Successful CI run:
https://github.com/viccuad/disallow-service-loadbalancer-policy/actions/runs/12352648538
Successful :latest release:
https://github.com/viccuad/disallow-service-loadbalancer-policy/actions/runs/12352648552
Sucessful failing release because artifacthub-pkg.yml has not been updated:
https://github.com/viccuad/disallow-service-loadbalancer-policy/actions/runs/12352705437/job/34470302880
Sucessful tagged release:
https://github.com/viccuad/disallow-service-loadbalancer-policy/actions/runs/12352738499

Additional Information

Tradeoff

Potential improvement

TODO:

Implement normal CI per policy on PRs (it's marked as a nice-to-have).
After approval, rebase and substitute the DROP commit, consuming my fork of kubewarden/github-actions for testing, and instead bump versions for release.

fabriziosestito · 2024-12-09T07:36:38Z

opa-installer/action.yml

    required: false
-    default: v0.32.0
+    default: v0.65.0


wow that was an old version, maybe we should use some updatecli here

Good point, opened #139.

fabriziosestito · 2024-12-09T09:39:14Z

policy-release/action.yml

 runs:
  using: "composite"
  steps:
-    -
-      name: Login to GitHub Container Registry
+    - id: calculate-version


This should go in the monorepo action instead.
We should also use a policy version variable here to make this agnostic.
The rationale is that we want to decouple the reusable actions from whatever logic we have in the other repositories.

We should also use a policy version variable here to make this agnostic.

It's not as simple, as the GH release needs to match the git tag that it was triggered with (and use the correct OCI_TAG), only when the trigger of this workflow was indeed from a tag (e.g: v0.1.0) instead of the "monorepo tag" (policy/v0.1.0 edited into 0.1.0).

Instead of again recalculating the version from the "monorepo tag", as we do in the monorepo workflow definition, here I just add a new, non-required policy-version input, that when present changes the behaviour.

I can take this out of the action, but that means passing that information through the reusable workflows, and potentially editing workflows of the policies we have, as that breaks backwards compatibility.

After a talk, I think you are right and the calculate_version step can be dropped. Let me see.

Added some comments that remove the calculate-version from the policy-release action, and expect this calculation to have been taken place in the workflows, by now making policy-release.inputs.policy-version required.
Hence, amended the workflows to always calculate-version and pass this value down the chain.

Since the workflows inputs haven't changed, this is still a backwards compatible fix WRT workflows API/ABI.

(If this is is approved, I will rebase the fixup commit and drop the "DROP" commit on the rebase as expected).

fabriziosestito · 2024-12-11T14:57:48Z

@viccuad note: DCO is missing

viccuad · 2024-12-11T16:28:48Z

Thanks! The DCO is missing on the fixup! commit, it will get removed with the rebase.

jvanz

LGTM

jvanz · 2024-12-12T13:20:44Z

.github/workflows/reusable-test-policy-rust.yml

@@ -40,7 +40,7 @@ jobs:
        run: echo "version=$(sed  -n 's,^version = \"\(.*\)\",\1,p' Cargo.toml)" >> $GITHUB_OUTPUT
        shell: bash
      - name: Check that artifacthub-pkg.yml is up-to-date
-        uses: kubewarden/github-actions/check-artifacthub@v3.3.5
+        uses: viccuad/github-actions/check-artifacthub@main


Just remember to update that for the kubewarden org repository

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

`policy-working-dir` defaults to `.` for backwards compat. `version` gets calculated depending if `policy-version` was set, or we have a tag that is the version, for backwards compat. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

`policy-working-dir` defaults to `.` for backwards compat. `version` was and continues to be set, either being a tag or coming from a `policy-version` (the second element of a tag as `PolicyName/v0.1.0`). This is because we need a version to be able to call `VERSION=foo make artifacthub-pkg.yml`. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

…y-rego `policy-working-dir` defaults to `.` for backwards compat. If `policy-version` is set, we use it for `version`. If not, we see if there's a valid version tag that triggered the workflow, and use that for `version` for backwards compat. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

`policy-working-dir` defaults to `.` for backwards compat. `policy-version` is new. If set, it's used, if not, we take the github.ref which is expected to be a tag, for backwards compat with non-monorepo policy releases. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

`policy-working-dir` defaults to `.` for backwards compat. Note that, while we do `git push --force`, monorepo release jobs run with concurrency 1. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

This is needed to run the unit tests of the monorepo opa policies. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

It may be that it doesn't exist even, like in the monorepo case. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

This makes releases named "<tag>", e.g: like https://github.com/kubewarden/policy-server/releases/tag/v1.19.0 Instead of releases named "Release <tag>", e.g: https://github.com/kubewarden/verify-image-signatures/releases/tag/v0.3.0 Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

By expecting as input of `policy-release` GHA, we push up the stack the calculations of it. policy-release `input.policy-version` will always be in the form of `0.1.0`. Since now it's required, add it to all reusable workflows that call it. No change on those workflows is needed, they were already calculating their versions in their `calculate-version` steps. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

viccuad · 2024-12-16T12:47:51Z

Rebased with a new calculate-version step, that now correctly sets the version regardless if it was run manually by passing a policy-version, it was triggered via git, or triggered via a branch for :latest, :latest-feat-foo.

Rerun the test workflow runs, which are passing, and updated their links in the PR body.

Now that policy-release.input.policy-version is required, always calculate-version. For the left side of the &&: policy-release will push `:latest` for main runs, and `:v0.1.0` for tags. Hence we don't need to check if the run was triggered by a tag. For the right side of the &&: We may want to push an artifact irrespectively if we want it published on artifacthub, hence we don't need to check for inputs.artifacthub. It may happen that for a run triggered without a tag, policy-version will be empty or a previous existent tag, but it will not get used by neither check-artifacthub nor policy-release anyways. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

viccuad · 2024-12-17T13:40:14Z

Found a bug when running tests from PRs in calculate-version step, please don't merge.

viccuad · 2024-12-17T14:01:08Z

Ready for merge.

The bug happened when, on PRs (not a run from branch nor a run from a tag), the calculate-version step was incorrectly going through the arm of tags. For normal repos this was fine, but for monorepos, it meant that it was taking a tag of PolicyName/v0.1.0 and generating a version=olicyName/v0.1.0 instead of version=0.1.0, which breaks make artifacthub-pkg.yml.

This is now fixed, see https://github.com/kubewarden/github-actions/compare/ffc0a258bac24630fede41f0f7c2de2f771eb9e1..0698fff8784455d9278bf36fc845a9f1989eb91e.

jvanz · 2024-12-18T13:17:19Z

policy-release/action.yml

      shell: bash
-      if: ${{ startsWith(github.ref, 'refs/tags/') }}
+      if: ${{ ! startsWith(github.ref, 'refs/heads/') }}
      env:
        COSIGN_EXPERIMENTAL: 1


As far as I can remember, we use this env to allow keyless signing. This is not necessary anymore. Therefore, we can remove this COSIGN_EXPERIMENTAL env var.

True! Done in c3c3a10.

Newer versions of cosign don't need this env var anymore. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

viccuad · 2024-12-18T15:28:15Z

Tagged in main as v3.4.0.

viccuad self-assigned this Dec 6, 2024

viccuad changed the title ~~Monorepo~~ feat: Support policies in folders for reusable workflows Dec 6, 2024

github-actions bot added the kind/enhancement label Dec 6, 2024

viccuad marked this pull request as ready for review December 6, 2024 15:19

viccuad requested a review from a team as a code owner December 6, 2024 15:19

viccuad force-pushed the monorepo branch 2 times, most recently from 251cbf0 to da6b4c2 Compare December 6, 2024 15:31

fabriziosestito reviewed Dec 9, 2024

View reviewed changes

fabriziosestito requested changes Dec 9, 2024

View reviewed changes

viccuad requested a review from fabriziosestito December 11, 2024 13:48

viccuad force-pushed the monorepo branch from ae43d94 to d335753 Compare December 11, 2024 13:50

fabriziosestito approved these changes Dec 11, 2024

View reviewed changes

jvanz approved these changes Dec 12, 2024

View reviewed changes

viccuad mentioned this pull request Dec 13, 2024

Duplicate of https://github.com/kubewarden/github-actions/pull/138 #140

Closed

viccuad added 11 commits December 16, 2024 13:43

chore: Reformat with prettier

4ec4f30

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

feat: Add policy-working-dir to push-artifacthub action

c085d31

`policy-working-dir` defaults to `.` for backwards compat. Note that, while we do `git push --force`, monorepo release jobs run with concurrency 1. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

deps: Bump opa to v0.65.0

02456bf

This is needed to run the unit tests of the monorepo opa policies. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

fix: Force remove artifacthub-pkg.yml in check-artifacthub

3aca576

It may be that it doesn't exist even, like in the monorepo case. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

docs: Document expected version better

6cc87e3

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

viccuad force-pushed the monorepo branch from d335753 to 1792a1e Compare December 16, 2024 12:46

viccuad requested a review from jvanz December 16, 2024 12:48

viccuad requested a review from fabriziosestito December 16, 2024 12:48

fabriziosestito approved these changes Dec 16, 2024

View reviewed changes

viccuad mentioned this pull request Dec 17, 2024

ci: Add release workflow kubewarden/rego-policies-library#2

Merged

viccuad added 2 commits December 17, 2024 14:39

chore(deps): Prepare for v3.4.0 tagging

0698fff

Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

viccuad marked this pull request as draft December 17, 2024 13:39

viccuad force-pushed the monorepo branch 3 times, most recently from ffc0a25 to 0698fff Compare December 17, 2024 13:42

viccuad marked this pull request as ready for review December 17, 2024 14:01

jvanz approved these changes Dec 18, 2024

View reviewed changes

chore: Remove unneeded COSIGN_EXPERIMENTAL env vars

c3c3a10

Newer versions of cosign don't need this env var anymore. Signed-off-by: Víctor Cuadrado Juan <vcuadradojuan@suse.de>

viccuad merged commit bcfa376 into kubewarden:main Dec 18, 2024
3 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: Support policies in folders for reusable workflows #138

feat: Support policies in folders for reusable workflows #138

viccuad commented Dec 6, 2024 •

edited

Loading

fabriziosestito Dec 9, 2024

viccuad Dec 9, 2024

fabriziosestito Dec 9, 2024

viccuad Dec 9, 2024 •

edited

Loading

viccuad Dec 9, 2024 •

edited

Loading

viccuad Dec 11, 2024 •

edited

Loading

fabriziosestito commented Dec 11, 2024

viccuad commented Dec 11, 2024

jvanz left a comment

jvanz Dec 12, 2024

viccuad commented Dec 16, 2024

viccuad commented Dec 17, 2024 •

edited

Loading

viccuad commented Dec 17, 2024

jvanz Dec 18, 2024 •

edited

Loading

viccuad Dec 18, 2024 •

edited

Loading

viccuad commented Dec 18, 2024

feat: Support policies in folders for reusable workflows #138

feat: Support policies in folders for reusable workflows #138

Conversation

viccuad commented Dec 6, 2024 • edited Loading

Description

Test

Additional Information

Tradeoff

Potential improvement

fabriziosestito Dec 9, 2024

Choose a reason for hiding this comment

viccuad Dec 9, 2024

Choose a reason for hiding this comment

fabriziosestito Dec 9, 2024

Choose a reason for hiding this comment

viccuad Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

viccuad Dec 9, 2024 • edited Loading

Choose a reason for hiding this comment

viccuad Dec 11, 2024 • edited Loading

Choose a reason for hiding this comment

fabriziosestito commented Dec 11, 2024

viccuad commented Dec 11, 2024

jvanz left a comment

Choose a reason for hiding this comment

jvanz Dec 12, 2024

Choose a reason for hiding this comment

viccuad commented Dec 16, 2024

viccuad commented Dec 17, 2024 • edited Loading

viccuad commented Dec 17, 2024

jvanz Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

viccuad Dec 18, 2024 • edited Loading

Choose a reason for hiding this comment

viccuad commented Dec 18, 2024

viccuad commented Dec 6, 2024 •

edited

Loading

viccuad Dec 9, 2024 •

edited

Loading

viccuad Dec 9, 2024 •

edited

Loading

viccuad Dec 11, 2024 •

edited

Loading

viccuad commented Dec 17, 2024 •

edited

Loading

jvanz Dec 18, 2024 •

edited

Loading

viccuad Dec 18, 2024 •

edited

Loading