OKD Update 4.13.0-0.okd-2023-10-28-065448 --> 4.14.0-0.okd-2024-01-06-084517

[eliassteiner]

Hello everyone,
I'm dealing with the following update:
OKD Update 4.13.0-0.okd-2023-10-28-065448 --> 4.14.0-0.okd-2024-01-06-084517
The update has started, but it gets stuck at 2/32 Operators after the start.
The problem is that the kube-apiserver cannot be updated (Status: Cannotupdate).
Message: KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.

Can anyone help me?

[melledouwsma]

Hi. Please run oc adm must-gather to provide some debugging information on the current state of your cluster. Create a compressed archive from the must-gather directory the command created and provide that file to this discussion (either via upload or a public file sharing service). You can find more about this here. Thanks!

[vrutkovs]

KubeletMinorVersionUpgradeable: Kubelet minor versions on 6 nodes will not be supported in the next OpenShift minor version upgrade.

This message means "you should not start another minor version upgrade right now", but it doesn't mean the current upgrade is stuck. We'd need a must-gather to tell more

[eliassteiner]

here is the must gather : https://1drv.ms/u/s!AtdalFn8wgcEn7w59Yp0Nz5-sWqbSw?e=4FFIKr
PW: OKDUpdateFailed

Thank you for your help

[vrutkovs]

I doesn't look stuck, KubeletMinorVersion_KubeletMinorVersionUnsupportedNextUpgrade is Upgradable status. Lets give it few more hours to complete

[melledouwsma]

Not sure. The kube-apiserver has started on kube-apiserver-master-02 but that's pending now for 4 hours. The initContainer is reporting a fatal glibc issue.

$ omg get po -n openshift-kube-apiserver -o wide
NAME                            READY  STATUS     RESTARTS  AGE
kube-apiserver-guard-master-01  1/1    Running    0         4h48m
kube-apiserver-guard-master-02  0/1    Running    0         4h48m
kube-apiserver-guard-master-03  1/1    Running    0         4h48m
kube-apiserver-master-01        5/5    Running    2         2d
kube-apiserver-master-02        0/5    Pending    0         4h45m
kube-apiserver-master-03        5/5    Running    2         2d

$ omg logs -n openshift-kube-apiserver kube-apiserver-master-02 -c setup

2024-01-09T12:22:02.396185245Z Fatal glibc error: CPU does not support x86-64-v2

Could this be a UBI 9 (and therefore RHEL 9) change in minimal CPU requirements? In that case "upgrading" the virtual CPU to something newer would be required to finish the upgrade.

[eliassteiner]

thank you it was thsi. i had to change the cpu type in proxmox from x86-64-v2 to x86-64-v3