Add upgrade test which starts from the earliest possible image

[jlebon]

E.g. boot the first stable release we did, have it go through every barrier up to the target cosa build, and then run some basic sanity-checks. We wouldn't do this on every CI run, but lower downstream/higher tier.

Would be interesting to somehow cache the upgrade state so we don't have to go through all the barriers each time.

[jlebon]

Would be interesting to somehow cache the upgrade state so we don't have to go through all the barriers each time.

I guess the simplest way to do this would be to run these tests in e.g. AWS, and have kola optimistically look for VM snapshots to boot rather than always starting from scratch. Then we have a job which adds a snapshot after each release?

Could be QEMU too. Make a snapshot of the qcow2, and then... upload it to S3?

[dustymabe]

since I'm digging into this can of worms I might as well try to write the test.

[dustymabe]

some PRs that should complete this work:

• tests/kola: add extended upgrade test fedora-coreos-config#2310
• tests/kola: a few updates for the upgrade test fedora-coreos-config#2335
• jobs: add kola-upgrade test fedora-coreos-pipeline#847

[dustymabe]

Closing this out now since the above PRs have merged.