upgrade 4.11.0-0.okd-2022-11-19-05003 to 4.12.0-0.okd-2023-03-18-084815 problem

[ascherbakov686]

Hi,

i have a problem when upgrade worker node, in machine-config-daemon log:

0407 08:03:56.003080    3215 update.go:2118] Disk currentConfig rendered-worker-71ed862718ff8f282cb2d8d010cdc1b5 overrides node's currentConfig annotation rendered-worker-45126adc1a22667fc08df363a7db1557
I0407 08:03:56.005027    3215 daemon.go:1567] Validating against current config rendered-worker-71ed862718ff8f282cb2d8d010cdc1b5
E0407 08:03:56.133510    3215 writer.go:200] Marking Degraded due to: unexpected on-disk state validating against rendered-worker-71ed862718ff8f282cb2d8d010cdc1b5: expected target osImageURL "quay.io/openshift/okd-content@sha256:125e94f63520330aa50e85bcfd55e429d3051498c9ff3936628edd0d4ea5696b", have "quay.io/openshift/okd-content@sha256:b00705c742911060ac4d24314db04377e4a686d495441397c97be0bd7fd17d68" ("5c8d0623f6a05c84a1acfb501b099de4ffd3de0feccd533c8b83f3f44737e510")

when i try to upgrade from shell with command

run/bin/machine-config-daemon pivot quay.io/openshift/okd-content@sha256:125e94f63520330aa50e85bcfd55e429d3051498c9ff3936628edd0d4ea5696b

was returned that:

I0322 18:17:48.747930   38388 workqueue.go:143] about to send work queue error: <nil>
I0322 18:17:48.806702   38371 rpm-ostree.go:208] Previous pivot: [quay.io/openshift/okd-content@sha256:d10fe7ed3ac8aa66ccd0bb7deee549dba5edd05a1dae57e52830d2e69ea3a7c5](http://quay.io/openshift/okd-content@sha256:d10fe7ed3ac8aa66ccd0bb7deee549dba5edd05a1dae57e52830d2e69ea3a7c5)
I0322 18:17:49.072318   38371 rpm-ostree.go:245] No com.coreos.ostree-commit label found in metadata! Inspecting...
I0322 18:17:49.072353   38371 rpm-ostree.go:394] Running captured: ostree refs --repo /run/mco-machine-os-content/os-content-3262537460/srv/repo
error: error running ostree refs --repo /run/mco-machine-os-content/os-content-3262537460/srv/repo: exit status 1
error: opening repo: opendir(/run/mco-machine-os-content/os-content-3262537460/srv/repo): No such file or directory

please help

[vrutkovs]

Why did the usual update failed in the first place? Perhaps ostree-finalize-staged.service failed to complete? Check if you have unstaged deployments in rpm-ostree status on the node. If a pending deployment exists, you can run ostree admin finalize-staged to run it now

Instead of machine-config-daemon pivot try rpm-ostree rebase --experimental "ostree-unverified-registry:quay.io/openshift/okd-content@sha256:125e94f63520330aa50e85bcfd55e429d3051498c9ff3936628edd0d4ea5696b

[ascherbakov686]

Why did the usual update failed in the first place? Perhaps ostree-finalize-staged.service failed to complete? Check if you have unstaged deployments in rpm-ostree status on the node. If a pending deployment exists, you can run ostree admin finalize-staged to run it now

Instead of machine-config-daemon pivot try rpm-ostree rebase --experimental "ostree-unverified-registry:quay.io/openshift/okd-content@sha256:125e94f63520330aa50e85bcfd55e429d3051498c9ff3936628edd0d4ea5696b

thank you, this helped, worker node is fully upgraded to destination version!