-
Notifications
You must be signed in to change notification settings - Fork 155
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
tests/kola: add extended upgrade test #2310
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks sane overall (thanks for writing it!). But I think ideally we'd rework this so that it works with the locally built artifacts. I've been thinking about merging the kola upgrade test into an external test, which is relevant here.
One thing we could do is add a new startingImage
YAML key, which can take e.g. previous
(for the current upgrade test) and first
(for this one). Then kola would implement the logic to resolve the version and fetch the image to boot (it already has most of the code for this). Then the external test could use injectContainer: true
to have access to the OCI archive.
tests/kola/upgrade/extended/test.sh
Outdated
# in the compose OSTree repo. | ||
if vereq $version $last_release; then | ||
systemctl stop zincati | ||
rpm-ostree rebase fedora-compose: |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Shouldn't we check here that the version we staged matches $target_version
? Or alternatively, we can make this rpm-ostree rebase fedora-compose: $target_version
. That way rpm-ostree itself errors out if the version somehow isn't in the compose repo.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
applied
tests/kola/upgrade/extended/test.sh
Outdated
timeout 60s journalctl -b 0 -f --no-tail -u zincati.service \ | ||
| grep --max-count=1 'proceeding to stage it' && \ | ||
/tmp/autopkgtest-reboot-prepare reboot && sleep infinity | ||
|
||
# If we make it here then there was no update; let's error. | ||
fatal "Updating the system stalled out on version: $version" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This looks cryptic. Why not
if ! timeout ... | grep ...; then
fatal "Updating the system stalled out on version: $version"
fi
/tmp/autopkgtest-reboot-prepare reboot
sleep infinity
?
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
I didn't think what I had was too bad, but I reworked it a bit now
9a4e55e
to
59c4b7a
Compare
No problem.
I support something better than this, but probably can't invest much time in it for now.
👍 |
tests/kola/upgrade/extended/test.sh
Outdated
max_version=38 # hardcode for now because `next` went from | ||
# 37.20230303.1.0 -> 38.20230310.1.0 -> 37.20230303.1.1. | ||
# Can drop once a new 38 commit is at the tip of `next`. |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Since we're about to run the next
build we can just hold this PR for 4-12 hours and I can drop this hack/workaround (I think).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
This is still there, did you mean to drop it?
266649a
to
3d7195c
Compare
This test will attempt to test an upgrade from a given starting point (assumed by the caller passing in a specific `cosa kola run --build=x.y.z`) all the way to the latest build that is staged to be released. The test is basic in that it essentially tests 1) updates work 2) boot works. An example invocation for this test would look like: ``` cosa buildfetch --stream=next --build=34.20210904.1.0 --artifact=qemu cosa decompress --build=34.20210904.1.0 cosa kola run --build=34.20210904.1.0 --tag extended-upgrade ``` You can then monitor the console.txt or journal.txt to watch the upgrades happen via `tail -f tmp/kola/ext.config.upgrade.extened/*/console.txt'. Another useful thing to do is to just monitor the major events: ``` $ tail -f tmp/kola/ext.config.upgrade.extended/*/journal.txt | grep --color -i 'ok reached version' Mar 20 02:25:12.584096 kola-runext-test.sh[2085]: ok Reached version: 34.20210904.1.0 Mar 20 02:26:37.107738 kola-runext-test.sh[3538]: ok Reached version: 35.20210924.1.0 Mar 20 02:28:08.509482 kola-runext-test.sh[1668]: ok Reached version: 36.20220418.1.0 Mar 20 02:28:41.242555 kola-runext-test.sh[1649]: ok Reached version: 36.20220906.1.0 Mar 20 02:29:33.635979 kola-runext-test.sh[1635]: ok Reached version: 37.20221111.1.0 Mar 20 02:30:16.657854 kola-runext-test.sh[1469]: ok Reached version: 37.20230303.1.0 ```
3d7195c
to
7d6d879
Compare
This test will attempt to test an upgrade from a given starting point (assumed by the caller passing in a specific
cosa kola run --build=x.y.z
) all the way to the latest build that is staged to be released. The test is basic in that it essentially tests 1) updates work 2) boot works.An example invocation for this test would look like:
You can then monitor the console.txt or journal.txt to watch the upgrades happen via
tail -f tmp/kola/ext.config.upgrade.extened/*/console.txt
.Another useful thing to do is to just monitor the major events: