mantle/kola/harness: allow success if all tests are denylisted

[Adam0Brien]

Fix : #3464

This PR introduces a new check in coreos-assembler that notifies users when running only denylisted tests

eg output:

adamobrien@fedora:~/cosa/fcos » cosa kola run podman.workflow                                                                                                                                                                             1 ↵
+cosa:16> podman run --rm -ti '--security-opt=label=disable' --privileged '--uidmap=1000:0:1' '--uidmap=0:1:1000' '--uidmap=1001:1001:64536' '-v=/home/adamobrien/cosa/fcos:/srv/' '--device=/dev/kvm' '--device=/dev/fuse' '--tmpfs=/tmp' '-v=/var/tmp:/var/tmp' '--name=cosa' '-v=/home/adamobrien/cosa/fedora-coreos-config/:/srv/src/config/:ro' '-v=/home/adamobrien/cosa/coreos-assembler/src/:/usr/lib/coreos-assembler/:ro' localhost/coreos-assembler kola run podman.workflow
kola -p qemu-unpriv run podman.workflow --output-dir tmp/kola
⚠️  Skipping kola test pattern "fcos.internet":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
⚠️  Skipping kola test pattern "podman.workflow":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
There are no tests to run because all tests are denylisted. Output in tmp/kola
+cosa:25> rc=0 
+cosa:25> set +x

And an additional feature that ensures your running a test with a valid name

adamobrien@fedora:~/cosa/fcos » cosa kola run test.that.does.not.exist
+cosa:16> podman run --rm -ti '--security-opt=label=disable' --privileged '--uidmap=1000:0:1' '--uidmap=0:1:1000' '--uidmap=1001:1001:64536' '-v=/home/adamobrien/cosa/fcos:/srv/' '--device=/dev/kvm' '--device=/dev/fuse' '--tmpfs=/tmp' '-v=/var/tmp:/var/tmp' '--name=cosa' '-v=/home/adamobrien/cosa/fedora-coreos-config/:/srv/src/config/:ro' '-v=/home/adamobrien/cosa/coreos-assembler/src/:/usr/lib/coreos-assembler/:ro' localhost/coreos-assembler kola run test.that.does.not.exist
kola -p qemu-unpriv run test.that.does.not.exist --output-dir tmp/kola
⚠️  Skipping kola test pattern "fcos.internet":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
⚠️  Skipping kola test pattern "podman.workflow":
  👉 https://github.com/coreos/coreos-assembler/pull/1478
2023-06-06T13:00:38Z kola: The user provided pattern didn't match any tests: test.that.does.not.exist
error: failed to execute cmd-kola: exit status 1
+cosa:25> rc=1 

[openshift-ci]

Skipping CI for Draft Pull Request.
If you want CI signal for your change, please convert it to an actual PR.
You can still manually trigger a test run with /test all

[dustymabe]

This looks good, but let's squash the commits down into a single commit if possible.

[dustymabe]

LGTM

[dustymabe]

Once this merges we should be able to merge coreos/fedora-coreos-config#2446

[dustymabe]

A few things I found later. Sorry.

Also, since we have to push up a new commit anyway maybe we can add some more context to the commit title/body:

mantle/kola/harness: allow success if all tests are denylisted

This will allow us to denylist certain categories of tests and not
have the pipeline fail. See [1] for a real world example of why this
is useful.

[1] https://github.com/coreos/fedora-coreos-config/commit/e983073e791073ae3e4fb5f4fc28f3d0e1e41ebb

[dustymabe]

/retest

…

On Fri, Jun 2, 2023, at 00:16, openshift-ci[bot] wrote: @Adam0Brien <https://github.com/Adam0Brien>: The following test *failed*, say `/retest` to rerun all failed tests or `/retest-required` to rerun all mandatory failed tests: Test name Commit Details Required Rerun command ci/prow/rhcos 270bd12 link <https://prow.ci.openshift.org/view/gs/origin-ci-test/pr-logs/pull/coreos_coreos-assembler/3489/pull-ci-coreos-coreos-assembler-main-rhcos/1664472418149732352> true `/test rhcos` Full PR test history <https://prow.ci.openshift.org/pr-history?org=coreos&repo=coreos-assembler&pr=3489>. Your PR dashboard <https://prow.ci.openshift.org/pr?query=is:pr+state:open+author:Adam0Brien>. Instructions for interacting with me using PR comments are available here <https://git.k8s.io/community/contributors/guide/pull-requests.md>. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra <https://github.com/kubernetes/test-infra/issues/new?title=Prow%20issue:> repository. I understand the commands that are listed here <https://go.k8s.io/bot-commands>. — Reply to this email directly, view it on GitHub <#3489 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/ABCR63T43SWPZF5JWYBDP4LXJFSIPANCNFSM6AAAAAAYSXLETM>. You are receiving this because you commented.Message ID: ***@***.***>

[Adam0Brien]

/retest

[dustymabe]

LGTM

[dustymabe]

/retest

[Adam0Brien]

/retest