kola: add gce support for --find-parent-image #1468

jlebon · 2020-05-21T16:30:56Z

We want to run upgrade tests in the pipeline now.

jlebon · 2020-05-21T16:54:20Z

(Didn't actually test this against GCP; need to dust off credentials -- or @dustymabe are you able to sanity check it?)

darkmuggle · 2020-05-21T17:07:27Z

LGTM. The CI failures seem to be a flake.

dustymabe · 2020-05-21T21:44:25Z

mantle/cmd/kola/kola.go

@@ -568,8 +568,13 @@ func syncFindParentImageOptions() error {
 		if err != nil {
 			return err
 		}
+	case "gce":


should this be gcp?

I think kola only knows gce right now. We should probably deprecate gce and move it to gcp.

dustymabe · 2020-05-21T22:09:41Z

ok I rebased this on top of #1335 over in dustymabe@aebe946

was going to try to run a test today but I think we have some chicken/egg problems here. Will discuss with @jlebon tomorrow more about how to properly test this.

dustymabe · 2020-05-22T15:37:02Z

#1335 merged so we should be able to rebase this now

jlebon · 2020-05-22T16:09:47Z

OK, updated this now and added a fallback for fedora-coreos-cloud!

darkmuggle

The CI bot is asking for my review.

LGTM, and based on the conversation this seems ready.

dustymabe · 2020-05-22T16:42:30Z

OK, updated this now and added a fallback for fedora-coreos-cloud!

I don't see a new upload from you (maybe my browswer window is acting up)?

We want to run upgrade tests in the pipeline now.

openshift-ci-robot · 2020-05-22T17:24:19Z

[APPROVALNOTIFIER] This PR is APPROVED

This pull-request has been approved by: darkmuggle, jlebon

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

~~OWNERS~~ [darkmuggle,jlebon]

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

jlebon · 2020-05-22T17:24:37Z

Yeah, GitHub is acting up. I just restamped the commit and re-pushed!

dustymabe · 2020-05-22T19:05:33Z

it works!

# cosa kola --upgrades --no-test-exit-error --platform=gce --gce-project=fedora-coreos-testing  --gce-json-key=/srv/fedora-coreos-testing-gcp-kola-tests.json
# kola --output-dir tmp/kola-upgrade run-upgrade --no-test-exit-error --platform=gce --gce-project=fedora-coreos-testing --gce-json-key=/srv/fedora-coreos-testing-gcp-kola-tests.json --qemu-image-dir tmp/kola-qemu-cache -v --find-parent-image
2020-05-22T19:00:20Z cli: Started logging at level INFO
2020-05-22T19:00:20Z cli: Started logging at level INFO
=== RUN   fcos.upgrade.basic
=== RUN   fcos.upgrade.basic/setup
=== RUN   fcos.upgrade.basic/upgrade-from-previous
=== RUN   fcos.upgrade.basic/upgrade-from-current
--- PASS: fcos.upgrade.basic (210.75s)
    --- PASS: fcos.upgrade.basic/setup (34.13s)
    --- PASS: fcos.upgrade.basic/upgrade-from-previous (49.00s)
    --- PASS: fcos.upgrade.basic/upgrade-from-current (40.52s)
PASS, output in tmp/kola-upgrade

dustymabe · 2020-05-22T19:06:14Z

CI failed with:

cluster.go:141: Error: unable to pull docker.io/library/nginx: unable to pull image: Error parsing image configuration: too many requests to registry

dustymabe

/lgtm

dustymabe · 2020-05-22T19:17:52Z

/retest sanity
/refresh

openshift-ci-robot · 2020-05-22T19:17:54Z

@dustymabe: The /retest command does not accept any targets.
The following commands are available to trigger jobs:

/test sanity

Use /test all to run all jobs.

In response to this:

/retest sanity
/refresh

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

dustymabe · 2020-05-22T19:18:16Z

/test sanity
/refresh

dustymabe · 2020-05-22T20:20:41Z

tests pass for me when I run them locally:

$ cosa kola run --parallel 4
COREOS_ASSEMBLER_CONTAINER_RUNTIME_ARGS=-v /var/b/shared/code/github.com/coreos/coreos-assembler//mantle/bin/:/usr/local/bin/:ro
COREOS_ASSEMBLER_GIT=/var/b/shared/code/github.com/coreos/coreos-assembler/
COREOS_ASSEMBLER_CONFIG_GIT=/var/b/shared/code/github.com/coreos/fedora-coreos-config/
+ podman run --rm -ti --security-opt label=disable --privileged --uidmap=1000:0:1 --uidmap=0:1:1000 --uidmap 1001:1001:64536 -v /var/b/shared/assembler/fedora:/srv/ --device /dev/kvm --device /dev/fuse --tmpfs /tmp -v /var/tmp:/var/tmp --name cosa -v /var/b/shared/code/github.com/coreos/fedora-c4
kola -p qemu-unpriv --output-dir tmp/kola run --parallel 4
=== RUN   coreos.selinux.boolean
=== RUN   podman.base
=== RUN   rhcos.selinux.boolean.persist
=== RUN   coreos.ignition.v2.users
=== RUN   ostree.remote
=== RUN   coreos.ignition.groups
=== RUN   podman.network-single
=== RUN   coreos.tls.fetch-urls
=== RUN   ext.config
=== RUN   coreos.ignition.once
=== RUN   fcos.users.shells
=== RUN   rpmostree.status
=== RUN   rootfs.uuid
=== RUN   coreos.auth.verify
=== RUN   coreos.selinux.enforce
=== RUN   rpmostree.install-uninstall
=== RUN   coreos.ignition.instantiated.enable-service
=== RUN   fcos.internet
=== RUN   fcos.filesystem
=== RUN   coreos.ignition.sethostname
=== RUN   coreos.ignition.journald-log
=== RUN   systemd.sysusers.gshadow
=== RUN   ostree.hotfix
=== RUN   coreos.ignition.failure
=== RUN   ostree.unlock
=== RUN   coreos.ignition.mount.partitions
=== RUN   basic
=== RUN   podman.workflow
=== RUN   coreos.ignition.mount.disks
=== RUN   coreos.ignition.resource.remote
=== RUN   rpmostree.upgrade-rollback
=== RUN   rpmostree.install-uninstall/install
--- PASS: coreos.ignition.journald-log (23.74s)
--- PASS: fcos.users.shells (23.98s)
--- PASS: coreos.selinux.boolean (43.60s)
--- PASS: coreos.auth.verify (33.52s)
=== RUN   fcos.filesystem/suid
=== RUN   fcos.filesystem/sgid
=== RUN   fcos.filesystem/writablefiles
=== RUN   fcos.filesystem/writabledirs
=== RUN   fcos.filesystem/stickydirs
=== RUN   fcos.filesystem/blacklist
--- PASS: coreos.selinux.enforce (43.51s)
--- PASS: fcos.filesystem (25.97s)
    --- PASS: fcos.filesystem/suid (0.66s)
    --- PASS: fcos.filesystem/sgid (0.40s)
    --- PASS: fcos.filesystem/writablefiles (0.31s)
    --- PASS: fcos.filesystem/writabledirs (0.33s)
    --- PASS: fcos.filesystem/stickydirs (0.39s)
    --- PASS: fcos.filesystem/blacklist (0.37s)
--- PASS: coreos.ignition.sethostname (24.24s)
--- PASS: coreos.ignition.groups (30.42s)
--- PASS: ext.config (26.28s)
=== RUN   rpmostree.install-uninstall/uninstall
--- PASS: coreos.ignition.once (50.15s)
--- PASS: coreos.tls.fetch-urls (24.23s)
--- PASS: coreos.ignition.v2.users (24.37s)
=== RUN   fcos.internet/PodmanEcho
--- PASS: rpmostree.install-uninstall (154.39s)
    --- PASS: rpmostree.install-uninstall/install (90.29s)
    --- PASS: rpmostree.install-uninstall/uninstall (39.93s)
=== RUN   fcos.internet/PodmanWgetHead
--- PASS: fcos.internet (38.48s)
    --- PASS: fcos.internet/PodmanEcho (3.09s)
    --- PASS: fcos.internet/PodmanWgetHead (0.91s)
=== RUN   ostree.remote/add
=== RUN   ostree.remote/list
=== RUN   ostree.remote/show-url
=== RUN   ostree.remote/refs
=== RUN   ostree.remote/summary
=== RUN   ostree.remote/delete
--- PASS: ostree.remote (27.45s)
    --- PASS: ostree.remote/add (0.20s)
    --- PASS: ostree.remote/list (0.15s)
    --- PASS: ostree.remote/show-url (0.15s)
    --- PASS: ostree.remote/refs (0.86s)
    --- PASS: ostree.remote/summary (1.85s)
    --- PASS: ostree.remote/delete (0.62s)
=== RUN   basic/DbusPerms
=== RUN   basic/NetworkScripts
=== RUN   basic/ServicesActive
=== RUN   basic/ReadOnly
=== RUN   basic/Useradd
=== RUN   basic/MachineID
=== RUN   basic/PortSSH
=== RUN   rpmostree.upgrade-rollback/upgrade
--- PASS: basic (26.08s)
    --- PASS: basic/DbusPerms (0.35s)
    --- PASS: basic/NetworkScripts (0.18s)
    --- PASS: basic/ServicesActive (0.15s)
    --- PASS: basic/ReadOnly (0.14s)
    --- PASS: basic/Useradd (0.40s)
    --- PASS: basic/MachineID (0.15s)
    --- PASS: basic/PortSSH (0.15s)
--- PASS: coreos.ignition.resource.remote (24.02s)
=== RUN   rpmostree.upgrade-rollback/rollback
=== RUN   rootfs.uuid/RandomUUID
--- PASS: rootfs.uuid (24.62s)
    --- PASS: rootfs.uuid/RandomUUID (0.20s)
--- PASS: rpmostree.upgrade-rollback (71.72s)
    --- PASS: rpmostree.upgrade-rollback/upgrade (23.58s)
    --- PASS: rpmostree.upgrade-rollback/rollback (24.50s)
--- PASS: coreos.ignition.mount.disks (50.08s)
=== RUN   podman.workflow/run
=== RUN   podman.workflow/exec
=== RUN   podman.workflow/stop
--- PASS: rpmostree.status (24.51s)
=== RUN   podman.workflow/remove
=== RUN   podman.workflow/delete
--- PASS: podman.workflow (36.73s)
    --- PASS: podman.workflow/run (10.30s)
            cluster.go:141: Trying to pull docker.io/library/nginx...
            cluster.go:141: Getting image source signatures
            cluster.go:141: Copying blob sha256:11fa52a0fdc084d7fc3bbcb774389fd37b148ee98e7829cea4af189735acf848
            cluster.go:141: Copying blob sha256:afb6ec6fdc1c3ba04f7a56db32c5ff5ff38962dc4cd0ffdef5beaa0ce2eb77e2
            cluster.go:141: Copying blob sha256:b90c53a0b69244e37b3f8672579fc3dec13293eeb574fa0fdddf02da1e192fd6
            cluster.go:141: Copying config sha256:9beeba249f3ee158d3e495a6ac25c5667ae2de8a43ac2a8bfd2bf687a58c06c9
            cluster.go:141: Writing manifest to image destination
            cluster.go:141: Storing signatures
    --- PASS: podman.workflow/exec (0.34s)
    --- PASS: podman.workflow/stop (1.02s)
    --- PASS: podman.workflow/remove (0.57s)
    --- PASS: podman.workflow/delete (0.86s)
            cluster.go:141: grep: docker.io/library/nginx:latest: No such file or directory
            cluster.go:141: bash: line 1: Deleted:: command not found
--- PASS: rhcos.selinux.boolean.persist (46.03s)
--- SKIP: coreos.ignition.failure (2.00s)
        qemufailure.go:50: Needs https://github.com/coreos/ignition-dracut/pull/146
--- PASS: coreos.ignition.instantiated.enable-service (24.51s)
=== RUN   podman.base/info
=== RUN   podman.base/resources
--- PASS: podman.base (35.41s)
    --- PASS: podman.base/info (0.70s)
    --- PASS: podman.base/resources (11.05s)
            cluster.go:141: Getting image source signatures
            cluster.go:141: Copying blob sha256:4315573f29c8a71dc5f0c142b5a8c26ff3c446aab840b731aac46ee9f8570aad
            cluster.go:141: Copying config sha256:be82222f2e85cf4d7ab68c85510b762b42616631e451d2d9aeceec0fa69a914d
            cluster.go:141: Writing manifest to image destination
            cluster.go:141: Storing signatures
            cluster.go:141: Your kernel does not support Block I/O weight or the cgroup is not mounted. Weight discarded.
=== RUN   ostree.hotfix/unlock
=== RUN   ostree.hotfix/install
=== RUN   ostree.hotfix/uninstall
=== RUN   ostree.hotfix/persist
=== RUN   ostree.unlock/unlock
=== RUN   ostree.unlock/install
=== RUN   ostree.unlock/uninstall
=== RUN   ostree.unlock/discard
--- PASS: podman.network-single (217.75s)
        cluster.go:141: Getting image source signatures
        cluster.go:141: Copying blob sha256:d54ea34165a1462070f1b19c4ea87bffe76e1f6ebeee881d8befc5b2c9d8989c
        cluster.go:141: Copying config sha256:c7abb3efb8da0004da6e062cd7c183699bb98eaaffddfe621622df8de1f502a8
        cluster.go:141: Writing manifest to image destination
        cluster.go:141: Storing signatures
=== RUN   ostree.hotfix/rollback
--- PASS: coreos.ignition.mount.partitions (50.44s)
--- PASS: ostree.unlock (45.22s)
    --- PASS: ostree.unlock/unlock (0.44s)
    --- PASS: ostree.unlock/install (0.67s)
    --- PASS: ostree.unlock/uninstall (0.57s)
    --- PASS: ostree.unlock/discard (20.06s)
--- PASS: systemd.sysusers.gshadow (24.12s)
--- PASS: ostree.hotfix (71.28s)
    --- PASS: ostree.hotfix/unlock (6.51s)
    --- PASS: ostree.hotfix/install (0.72s)
    --- PASS: ostree.hotfix/uninstall (0.58s)
    --- PASS: ostree.hotfix/persist (19.23s)
    --- PASS: ostree.hotfix/rollback (20.82s)
PASS, output in tmp/kola
+ rc=0
+ set +x

cgwalters · 2020-05-22T20:23:33Z

tests pass for me when I run them locally:

Yeah, let's just do
/override continuous-integration/jenkins/pr-merge
for now

openshift-ci-robot · 2020-05-22T20:23:36Z

@cgwalters: Overrode contexts on behalf of cgwalters: continuous-integration/jenkins/pr-merge

In response to this:

tests pass for me when I run them locally:

Yeah, let's just do
/override continuous-integration/jenkins/pr-merge
for now

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

openshift-ci-robot requested review from darkmuggle and LorbusChris May 21, 2020 16:30

openshift-ci-robot added the approved label May 21, 2020

This was referenced May 21, 2020

Jenkinsfile.kola.gcp: drop --gce-image coreos/fedora-coreos-pipeline#240

Merged

kola tests fail in GCP environment when running in pipeline coreos/fedora-coreos-tracker#487

Closed

dustymabe reviewed May 21, 2020

View reviewed changes

kola: fix error-handling for --find-parent-image

a582590

darkmuggle approved these changes May 22, 2020

View reviewed changes

kola: add gce support for --find-parent-image

beccce3

We want to run upgrade tests in the pipeline now.

jlebon force-pushed the pr/add-gce-upgrade-tests branch from 6ba4f94 to beccce3 Compare May 22, 2020 17:24

dustymabe approved these changes May 22, 2020

View reviewed changes

dustymabe merged commit 3b691fb into coreos:master May 22, 2020

jlebon deleted the pr/add-gce-upgrade-tests branch July 6, 2020 20:31

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

kola: add gce support for --find-parent-image #1468

kola: add gce support for --find-parent-image #1468

jlebon commented May 21, 2020

jlebon commented May 21, 2020

darkmuggle commented May 21, 2020

dustymabe May 21, 2020

jlebon May 22, 2020

dustymabe commented May 21, 2020

dustymabe commented May 22, 2020

jlebon commented May 22, 2020

darkmuggle left a comment

dustymabe commented May 22, 2020

openshift-ci-robot commented May 22, 2020

jlebon commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe left a comment

dustymabe commented May 22, 2020

openshift-ci-robot commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe commented May 22, 2020

cgwalters commented May 22, 2020

openshift-ci-robot commented May 22, 2020

kola: add gce support for --find-parent-image #1468

kola: add gce support for --find-parent-image #1468

Conversation

jlebon commented May 21, 2020

jlebon commented May 21, 2020

darkmuggle commented May 21, 2020

dustymabe May 21, 2020

Choose a reason for hiding this comment

jlebon May 22, 2020

Choose a reason for hiding this comment

dustymabe commented May 21, 2020

dustymabe commented May 22, 2020

jlebon commented May 22, 2020

darkmuggle left a comment

Choose a reason for hiding this comment

dustymabe commented May 22, 2020

openshift-ci-robot commented May 22, 2020

jlebon commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe left a comment

Choose a reason for hiding this comment

dustymabe commented May 22, 2020

openshift-ci-robot commented May 22, 2020

dustymabe commented May 22, 2020

dustymabe commented May 22, 2020

cgwalters commented May 22, 2020

openshift-ci-robot commented May 22, 2020