Upgrade from 4.10.0-0.okd-2022-07-09-073606 to 4.11.0-0.okd-2023-01-14-152430 fail at machine_config operator #1471

KevinChu148 · 2023-01-18T04:42:01Z

KevinChu148
Jan 18, 2023

Hi all,

I upgrade from 4.10.0-0.okd-2022-07-09-073606 to 4.11.0-0.okd-2023-01-14-152430 fail at machine_config operator.

My implementation process:

-Review the available cluster updates
$ oc adm upgrade

4.11.0-0.okd-2022-07-29-154152 registry.ci.openshift.org/origin/release@sha256:a506d80873f7c51201c614be9c8defa6455f3b593faa54d11ded7b4d20c0fc4e
4.11.0-0.okd-2022-08-20-022919 registry.ci.openshift.org/origin/release@sha256:7a99c7af2f3e6107107a6cf92669fef74ebbd0cc1780fbd187be2410b003a429
4.11.0-0.okd-2022-10-28-153352 registry.ci.openshift.org/origin/release@sha256:13fceee2e4e919d942b8252e8a86b1bd62371cb88cfcb45746d6f232a7b9acba
4.11.0-0.okd-2022-11-19-050030 registry.ci.openshift.org/origin/release@sha256:b2e925d0178130baa99ebdcdf9bfe124a1fa496cae955af5543652c2166def63
4.11.0-0.okd-2023-01-14-152430 registry.ci.openshift.org/origin/release@sha256:27c99f5b7734f773a7981b9e527417edcda0eee217117632198d64aa33e95c46

$ oc adm upgrade --to=4.11.0-0.okd-2023-01-14-152430 --allow-upgrade-with-warnings

Updating to 4.11.0-0.okd-2023-01-14-152430

Operator updated: 31/32 operator (last operator need update: machine_config)

I wait 1 master node "Ready" for a longtime. But over 3 hours, the node still "Ready,SchedulingDisabled" status and still old version "v1.23.5+3afdacb"

$ oc get node
NAME                                     STATUS                     ROLES    AGE    VERSION
okd4-compute-1.tanhung.okd.local         Ready                      worker   226d   v1.24.6+5658434
okd4-compute-2.tanhung.okd.local         Ready                      worker   226d   v1.24.6+5658434
okd4-control-plane-1.tanhung.okd.local   Ready                      master   226d   v1.24.6+5658434
okd4-control-plane-2.tanhung.okd.local   Ready                      master   226d   v1.24.6+5658434
okd4-control-plane-3.tanhung.okd.local   Ready,SchedulingDisabled   master   226d   v1.23.5+3afdacb

Other nodes upgraded to "v1.24.6+5658434" version

$ oc get co
NAME                            VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication                  4.11.0-0.okd-2023-01-14-152430   False       False         True       31m     OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.tanhung.okd.local/healthz": EOF
baremetal                       4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
cloud-controller-manager        4.11.0-0.okd-2023-01-14-152430   True        False         False      3d13h
cloud-credential                4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
cluster-autoscaler              4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
config-operator                 4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
console                         4.11.0-0.okd-2023-01-14-152430   False       False         True       31m     RouteHealthAvailable: failed to GET route (https://console-openshift-console.apps.tanhung.okd.local): Get "https://console-openshift-console.apps.tanhung.okd.local": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
csi-snapshot-controller         4.11.0-0.okd-2023-01-14-152430   True        False         False      35h
dns                             4.11.0-0.okd-2023-01-14-152430   True        False         False      3d13h
etcd                            4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
image-registry                  4.11.0-0.okd-2023-01-14-152430   True        False         False      32m
ingress                         4.11.0-0.okd-2023-01-14-152430   True        False         False      32m
insights                        4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
kube-apiserver                  4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
kube-controller-manager         4.11.0-0.okd-2023-01-14-152430   True        False         True       226d    GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": net/http: TLS handshake timeout
kube-scheduler                  4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
kube-storage-version-migrator   4.11.0-0.okd-2023-01-14-152430   True        False         False      32m
machine-api                     4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
machine-approver                4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
machine-config                  4.10.0-0.okd-2022-07-09-073606   True        True          True       2d21h   Unable to apply 4.11.0-0.okd-2023-01-14-152430: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, pool master has not progressed to latest configuration: controller version mismatch for rendered-master-6143964571dd13f34c856f07a62d1f0a expected b36482885ba1304e122e7c01c26cd671dfdd0418 has c3ac7f07f1cae32456a7ab361e29f49af7eb0802: 2 (ready 2) out of 3 nodes are updating to latest configuration rendered-master-e32b2175ac54bf7c34fbd30768f5040e, retrying]
marketplace                     4.11.0-0.okd-2023-01-14-152430   True        False         False      198d
monitoring                      4.11.0-0.okd-2023-01-14-152430   False       True          True       20m     Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
network                         4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
node-tuning                     4.11.0-0.okd-2023-01-14-152430   True        False         False      67m
openshift-apiserver             4.11.0-0.okd-2023-01-14-152430   False       False         False      16m     APIServicesAvailable: "apps.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "authorization.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "image.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "project.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "quota.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "route.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "security.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
APIServicesAvailable: "template.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
openshift-controller-manager               4.11.0-0.okd-2023-01-14-152430   True        False         False      64m
openshift-samples                          4.11.0-0.okd-2023-01-14-152430   True        False         False      68m
operator-lifecycle-manager                 4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
operator-lifecycle-manager-catalog         4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
operator-lifecycle-manager-packageserver   4.11.0-0.okd-2023-01-14-152430   False       True          False      16m     ClusterServiceVersion openshift-operator-lifecycle-manager/packageserver observed in phase Failed with reason: ComponentUnhealthy, message: apiServices not installed
service-ca                                 4.11.0-0.okd-2023-01-14-152430   True        False         False      226d
storage                                    4.11.0-0.okd-2023-01-14-152430   True        False         False      226d

I did it again and again and the error is still the same.

vrutkovs · 2023-01-18T06:26:47Z

vrutkovs
Jan 18, 2023
Maintainer

Please attach (or upload to the public file sharing service) must-gather archive

0 replies

KevinChu148 · 2023-01-18T12:21:15Z

KevinChu148
Jan 18, 2023
Author

I just upgrade again. The error is still the same. And here is my logs:
gather_network_logs.tar.gz
openshift-apiserver-audit.tar.gz

And file gather_audit_logs.tar.gz very big. I upload it to my google drive: https://drive.google.com/file/d/1cvoIq6VBnXkQ9WPdDdn92szcRAEvTEj6/view?usp=sharing

12 replies

vrutkovs Jan 19, 2023
Maintainer

Folks, please, documentation is not just "copy paste instructions to command line", it says "Alternatively" and "For example" there

KevinChu148 Jan 19, 2023
Author

ok. I ran: "oc adm must-gather". And here is my log:
must-gather.local.7506780661947579075.tar.gz
logrun.log

To collect "must-gather log" is very difficult. Due to the pod is destroyed continuously.

vrutkovs Jan 19, 2023
Maintainer

Machine controller is unable to evict pods on the third control plane node.

drain_controller.go:139] node okd4-control-plane-3.tanhung.okd.local: initiating drain
...
2023-01-19T13:50:47.909450460Z E0119 13:50:47.909418 1 drain_controller.go:110] error when evicting pods/"console-5b8fd5786b-m5wn9" -n "openshift-console" (will retry after 5s): Cannot evict pod as it would violate the pod's disruption budget.

Check your loadbalancer setup and try rebooting the third node?

KevinChu148 Jan 19, 2023
Author

After reboot third node. The error is the same.

# oc get node
NAME                                     STATUS                     ROLES    AGE    VERSION
okd4-compute-1.tanhung.okd.local         Ready                      worker   230d   v1.24.6+5658434
okd4-compute-2.tanhung.okd.local         Ready                      worker   230d   v1.24.6+5658434
okd4-control-plane-1.tanhung.okd.local   Ready                      master   230d   v1.24.6+5658434
okd4-control-plane-2.tanhung.okd.local   Ready                      master   230d   v1.24.6+5658434
okd4-control-plane-3.tanhung.okd.local   Ready,SchedulingDisabled   master   230d   v1.23.5+3afdacb

third node still "v1.23.5+3afdacb" version

# oc get co
NAME             VERSION                          AVAILABLE   PROGRESSING   DEGRADED   SINCE   MESSAGE
authentication   4.11.0-0.okd-2023-01-14-152430   False       False         True       4h15m   APIServicesAvailable: "oauth.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
OAuthServerRouteEndpointAccessibleControllerAvailable: Get "https://oauth-openshift.apps.tanhung.okd.local/healthz": EOF
OAuthServerServiceEndpointAccessibleControllerAvailable: Get "https://172.30.37.249:443/healthz": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
baremetal                  4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
cloud-controller-manager   4.11.0-0.okd-2023-01-14-152430   True        False         False      6d23h
cloud-credential           4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
cluster-autoscaler         4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
config-operator            4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
console                    4.11.0-0.okd-2023-01-14-152430   False       False         False      4h15m   DeploymentAvailable: 0 replicas available for console deployment
RouteHealthAvailable: route not yet available, https://console-openshift-console.apps.tanhung.okd.local returns '503 Service Unavailable'
csi-snapshot-controller                    4.11.0-0.okd-2023-01-14-152430   True        False         False      4d21h
dns                                        4.11.0-0.okd-2023-01-14-152430   True        False         False      6d22h
etcd                                       4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
image-registry                             4.11.0-0.okd-2023-01-14-152430   True        False         False      4h15m
ingress                                    4.11.0-0.okd-2023-01-14-152430   True        False         False      4h15m
insights                                   4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
kube-apiserver                             4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
kube-controller-manager                    4.11.0-0.okd-2023-01-14-152430   True        False         True       230d    GarbageCollectorDegraded: error fetching rules: Get "https://thanos-querier.openshift-monitoring.svc:9091/api/v1/rules": net/http: TLS handshake timeout
kube-scheduler                             4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
kube-storage-version-migrator              4.11.0-0.okd-2023-01-14-152430   True        False         False      4h16m
machine-api                                4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
machine-approver                           4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
machine-config                             4.10.0-0.okd-2022-07-09-073606   True        True          True       6d7h    Unable to apply 4.11.0-0.okd-2023-01-14-152430: error during syncRequiredMachineConfigPools: [timed out waiting for the condition, error pool master is not ready, retrying. Status: (pool degraded: true total: 3, ready 2, updated: 2, unavailable: 1)]
marketplace                                4.11.0-0.okd-2023-01-14-152430   True        False         False      201d
monitoring                                 4.11.0-0.okd-2023-01-14-152430   False       False         True       4h5m    Rollout of the monitoring stack failed and is degraded. Please investigate the degraded status error.
network                                    4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
node-tuning                                4.11.0-0.okd-2023-01-14-152430   True        False         False      4h49m
openshift-apiserver                        4.11.0-0.okd-2023-01-14-152430   False       False         True       4m43s   APIServicesAvailable: "build.openshift.io.v1" is not ready: an attempt failed with statusCode = 503, err = the server is currently unable to handle the request
openshift-controller-manager               4.11.0-0.okd-2023-01-14-152430   True        False         False      4h49m
openshift-samples                          4.11.0-0.okd-2023-01-14-152430   True        False         False      4h50m
operator-lifecycle-manager                 4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
operator-lifecycle-manager-catalog         4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
operator-lifecycle-manager-packageserver   4.11.0-0.okd-2023-01-14-152430   True        False         False      4m32s
service-ca                                 4.11.0-0.okd-2023-01-14-152430   True        False         False      230d
storage                                    4.11.0-0.okd-2023-01-14-152430   True        False         False      230d

imdmahajankanika Apr 18, 2023

@KevinChu148 Were you able to resolve the above error, as I am also getting the same

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Upgrade from 4.10.0-0.okd-2022-07-09-073606 to 4.11.0-0.okd-2023-01-14-152430 fail at machine_config operator #1471

{{title}}

{{editor}}'s edit

{{editor}}'s edit

Replies: 2 comments 12 replies

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

{{title}}

Select a reply

Upgrade from 4.10.0-0.okd-2022-07-09-073606 to 4.11.0-0.okd-2023-01-14-152430 fail at machine_config operator #1471

KevinChu148 Jan 18, 2023

Replies: 2 comments · 12 replies

vrutkovs Jan 18, 2023 Maintainer

KevinChu148 Jan 18, 2023 Author

vrutkovs Jan 19, 2023 Maintainer

KevinChu148 Jan 19, 2023 Author

vrutkovs Jan 19, 2023 Maintainer

KevinChu148 Jan 19, 2023 Author

imdmahajankanika Apr 18, 2023

KevinChu148
Jan 18, 2023

Replies: 2 comments 12 replies

vrutkovs
Jan 18, 2023
Maintainer

KevinChu148
Jan 18, 2023
Author

vrutkovs Jan 19, 2023
Maintainer

KevinChu148 Jan 19, 2023
Author

vrutkovs Jan 19, 2023
Maintainer

KevinChu148 Jan 19, 2023
Author