[RFE] kola should support to start previous build to do external tests

[HuijingHei]

Feature Request

Get suggestion by @jlebon , see coreos/rpm-ostree#4806 (comment):

Kola also has support for downloading the previous image, but as part of the run-upgrade suite; I think we should really fold that back into kola run proper and support external tests.

Desired Feature

To test rpm-ostree upgrade on old build, should download the previous build and start from it.

Add --from-previous option to download previous build and start, instead of the new built image. For example:
kola run --from-previous ext.rpm-ostree.test

Example Usage

To test rpm-ostree upgrade on previous build, can run with kola run --from-previous ext.rpm-ostree.test, in the test should upgrade rpm-ostree first, reboot and do tests.

Other Information

[HuijingHei]

@jlebon not sure my understanding is correct, could you help to review when you get a chance? Thanks!

[dustymabe]

I feel like there's a lot of overlap here with cosa buildfetch --build=xyz --stream=testing-devel && cosa kola run --build=xyz.

[jlebon]

Add --from-previous option to download previous build and start, instead of the new built image.

The way I was thinking about this was that it'd be another field in the test metadata like from: ..., and then we could support different syntaxes like from: previous (last build on that same stream), from: STREAM (latest build on STREAM), or from: STREAM@VERSION.

The key thing here is that the test is just like any other in the list and is run as part of the same kola run invocation (in practice, we'll probably want upgrade tests that use anything other than from: previous to be gated behind a requiredTag).

I feel like there's a lot of overlap here with cosa buildfetch --build=xyz --stream=testing-devel && cosa kola run --build=xyz.

Yeah, definitely. That's basically what we do today. This is an RFE to fold it into kola and make the UX even nicer (e.g. that'd let us simplify the kola() step a lot and ideally make it easier to run the extended upgrade test locally). Probably not high-priority but also not too hard since we already have the main pieces there.

[HuijingHei]

Sorry for the misunderstood, does this mean: add from: ... to kola.json like following?

#!/bin/bash
set -xeuo pipefail
## kola:
##   from: previous / STREAM / STREAM@VERSION
##   tags: needs-internet
...