Upgrade from 4.9.0-0.okd-2022-02-12-140851 to 4.10 hangs on (79% complete), waiting on marketplace

[vinisman]

Hello!
I began updating cluster from 4.9.0-0.okd-2022-02-12-140851 to 4.10.0
Stream: https://origin-release.apps.ci.l2s4.p1.openshiftapps.com/graph
Chanel: stable-4

But it cant finish, i have only one warning on [kube-apiserver] operator:
Preconditions failed for payload loaded version="4.10.0-0.okd-2022-06-10-131327" image="registry.ci.openshift.org/origin/release@sha256:ed37a14e619f59dc75c3beab12ebf0f577b3edd07fe126624ac295163b59faaa": Precondition "ClusterVersionUpgradeable" failed because of "KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade": Cluster operator kube-apiserver should not be upgraded between minor versions: KubeletMinorVersionUpgradeable: Kubelet minor versions on 20 nodes will not be supported in the next OpenShift minor version upgrade.
and no one node was overloaded.
So i have partial update and cant't understand what to do

% oc get mcp
NAME     CONFIG                                             UPDATED   UPDATING   DEGRADED   MACHINECOUNT   READYMACHINECOUNT   UPDATEDMACHINECOUNT   DEGRADEDMACHINECOUNT   AGE
infra    rendered-infra-fbe03e887b432e8ed458c2397f0f6970    True      False      False      3              3                   3                     0                      432d
master   rendered-master-2d315d879ffe0192ba765093cc0322ea   True      False      False      3              3                   3                     0                      457d
worker   rendered-worker-fbe03e887b432e8ed458c2397f0f6970   True      False      False      14             14                  14                    0                      457d

oc get mcp -o json | jq -r '.items[] | "\(.metadata.name) - paused: \(.spec.paused)"'
infra - paused: false
master - paused: false
worker - paused: false

[vinisman]

Also i found the same issue on bugzilla with no desigion
https://bugzilla.redhat.com/show_bug.cgi?id=2084531

[vrutkovs]

Please attach (or upload to the public file sharing service) must-gather archive

[vinisman]

Hi @vrutkovs!
Please look at must-gather logs https://drive.google.com/file/d/13vCJT4BdhbiycEqWXRlII-oJ3h-cPBwQ/view?usp=share_link

[vrutkovs]

MCO operator can't connect to two MCD pods:

2023-01-30T06:37:17.478148765Z I0130 06:37:17.478049       1 event.go:282] Event(v1.ObjectReference{Kind:"", Namespace:"", Name:"machine-config", UID:"d7aaefad-2480-4daf-b75c-4e6e17d7ce47", APIVersion:"", ResourceVersion:"", FieldPath:""}): type: 'Warning' reason: 'OperatorDegraded: MachineConfigDaemonFailed' Failed to resync 4.9.0-0.okd-2022-02-12-140851 because: timed out waiting for the condition during waitForDaemonsetRollout: Daemonset machine-config-daemon is not ready. status: (desired: 20, updated: 20, ready: 18, unavailable: 2)

Daemonset is fine however, try killing machine-config-operator-6b957fb4b-m8cm2 pod in openshift-machine-config-operator namespace?

[vinisman]

Kill but nothing has changed

[vinisman]

Any ideas? May be i can run upgrade using option --allow-upgrade-with-warnings=true ?

[vinisman]

Hi @vrutkovs ! Can you advice which way i can upgrade cluster ?

[vinisman]

Some additional info

NAME      VERSION                         AVAILABLE   PROGRESSING   SINCE   STATUS
version   4.9.0-0.okd-2022-02-12-140851   True        True          29d     Working towards 4.10.0-0.okd-2022-06-10-131327: 615 of 778 done (79% complete), waiting on marketplace

Log from marketplace-operator pod:

time="2023-02-27T15:44:49Z" level=info msg="Go Version: go1.16.9"
time="2023-02-27T15:44:49Z" level=info msg="Go OS/Arch: linux/amd64"
time="2023-02-27T15:44:49Z" level=info msg="[metrics] Registering marketplace metrics"
time="2023-02-27T15:44:49Z" level=info msg="[metrics] Serving marketplace metrics"
time="2023-02-27T15:44:49Z" level=info msg="TLS keys set, using https for metrics"
time="2023-02-27T15:44:49Z" level=info msg="Config API is available"
time="2023-02-27T15:44:49Z" level=info msg="setting up scheme"
I0227 15:44:50.974732       1 request.go:668] Waited for 1.03526226s due to client-side throttling, not priority and fairness, request: GET:https://172.30.0.1:443/apis/operators.coreos.com/v2?timeout=32s
time="2023-02-27T15:44:52Z" level=info msg="setting up health checks"
I0227 15:44:52.642859       1 leaderelection.go:243] attempting to acquire leader lease openshift-marketplace/marketplace-operator-lock...
I0227 15:44:52.662885       1 leaderelection.go:253] successfully acquired lease openshift-marketplace/marketplace-operator-lock
time="2023-02-27T15:44:52Z" level=info msg="became leader: marketplace-operator-7964f694fb-qp4bn"
time="2023-02-27T15:44:52Z" level=info msg="registering components"
time="2023-02-27T15:44:52Z" level=info msg="setting up the marketplace clusteroperator status reporter"
time="2023-02-27T15:44:55Z" level=info msg="setting up controllers"
time="2023-02-27T15:44:55Z" level=info msg="ensuring the default catalogsource resources"
time="2023-02-27T15:44:58Z" level=info msg="starting the marketplace clusteroperator status reporter"
time="2023-02-27T15:44:58Z" level=info msg="starting manager"
time="2023-02-27T15:44:58Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:44:58Z" level=info msg="Reconciling ConfigMap openshift-marketplace/marketplace-trusted-ca"
time="2023-02-27T15:44:58Z" level=info msg="[ca] Certificate Authorization ConfigMap openshift-marketplace/marketplace-trusted-ca is in sync with disk." name=marketplace-trusted-ca type=ConfigMap
time="2023-02-27T15:44:58Z" level=info msg="Reconciling OperatorHub cluster"
time="2023-02-27T15:44:58Z" level=error msg="[defaults] Error processing CatalogSource certified-operators - CatalogSource.operators.coreos.com \"certified-operators\" not found"
time="2023-02-27T15:44:58Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2023-02-27T15:44:58Z" level=error msg="[defaults] Error processing CatalogSource redhat-marketplace - CatalogSource.operators.coreos.com \"redhat-marketplace\" not found"
time="2023-02-27T15:44:58Z" level=error msg="[defaults] Error processing CatalogSource redhat-operators - CatalogSource.operators.coreos.com \"redhat-operators\" not found"
time="2023-02-27T15:45:18Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:45:38Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:45:58Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:46:18Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:46:38Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:46:58Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:47:18Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:47:38Z" level=info msg="[status] Previous and current ClusterOperator Status are the same, the ClusterOperator Status will not be updated."
time="2023-02-27T15:47:47Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"
time="2023-02-27T15:47:58Z" level=info msg="[defaults] CatalogSource community-operators is annotated and its spec is the same as the default spec"

[vinisman]

We found a problem, one of the operators can't reboot properly and deletinig the finalazier in deployment of this operator help us)