kola: support tags for --allow-rerun-success

[nikita-dubrovskii]

New format is:

--allow-rerun-success tags=tag1[,tag2]

To allow all tests simply:

--allow-rerun-success tags=all

Issue: coreos/fedora-coreos-pipeline#842

[dustymabe]

LGTM.

Did this work well in local testing? I ran some brief tests but I didn't spend a lot of time on it.

[dustymabe]

I think we need to pair this with a PR to the pipeline to fixup the calls there and I think we'll also need to backport this change to older COSA branches.

One other thing maybe we should do here is have --allow-rerun-success tags=xyz imply --rerun so the user doesn't have to specify both.

@jlebon - WDYT?

[nikita-dubrovskii]

LGTM.

Did this work well in local testing? I ran some brief tests but I didn't spend a lot of time on it.

yep, tested locally with enabling/disabling host's network during tests. Here is an example:

$cosa kola run --rerun --allow-rerun-success tags=foo,needs-internet  ext.config.rpm-ostree-countme
$cosa kola run --rerun --allow-rerun-success tags=all  ext.config.rpm-ostree-countme

Output looks good:

=== RUN   ext.config.rpm-ostree-countme
....
--- FAIL: ext.config.rpm-ostree-countme (25.78s)
        cluster.go:162: Error: Unit kola-runext.service exited with code 1
        cluster.go:162: 2023-05-02T14:14:00Z cli: Unit kola-runext.service exited with code 1
        harness.go:1147: kolet failed: : kolet run-test-unit failed: Process exited with status 1
FAIL, output in tmp/kola
...
======== Re-running failed tests (flake detection) ========
--- PASS: ext.config.rpm-ostree-countme (28.55s)
PASS, output in tmp/kola/rerun
+ rc=0

[dustymabe]

SGTM.. ok I don't think we need to backport this before merging because the only tests in the pipeline using --allow-rerun-success aren't run downstream right now. We just need to get the existing uses in the pipeline updated. I opened coreos/fedora-coreos-pipeline#868 for that.

[jlebon]

One other thing maybe we should do here is have --allow-rerun-success tags=xyz imply --rerun so the user doesn't have to specify both.

Yeah, that seems reasonable to me. It's not strictly new to this PR so I wouldn't block on it.