Update from Okd 4.11 to 4.12

[sahaniarun]

Appreciate community to share your experience on updating to latest OKD 4.12

Arun

[tdemaret]

Hey,

During the update, I have an issue with kubelet.service, I have incorrect syntax in ExecStart :

~#` cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target
Requires=crio.service kubelet-auto-node-size.service
After=network-online.target crio.service kubelet-auto-node-size.service
After=ostree-finalize-staged.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
ExecStartPre=/bin/rm -f /var/lib/kubelet/memory_manager_state
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env
EnvironmentFile=/etc/node-sizing.env

ExecStart=/usr/local/bin/kubenswrapper \
    /usr/bin/kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/control-plane,node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --address=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --cloud-provider=external \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
       \
      --hostname-override=${KUBELET_NODE_NAME} \
      --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
      --pod-infra-container-image=quay.io/openshift/okd-content@sha256:b28820e1027f7fe8e97430fcc7e91ea11b250f4b23524c6b2e252c81d2cc9903 \
      --system-reserved=cpu=${SYSTEM_RESERVED_CPU},memory=${SYSTEM_RESERVED_MEMORY},ephemeral-storage=${SYSTEM_RESERVED_ES} \
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target``

I have fix to override kubelet.service file. I am not sure, if I am alone with this case.

Regards,

Thibaut

[dkoci]

Hey,

During the update, I have an issue with kubelet.service, I have incorrect syntax in ExecStart :

~#` cat /etc/systemd/system/kubelet.service
[Unit]
Description=Kubernetes Kubelet
Wants=rpc-statd.service network-online.target
Requires=crio.service kubelet-auto-node-size.service
After=network-online.target crio.service kubelet-auto-node-size.service
After=ostree-finalize-staged.service

[Service]
Type=notify
ExecStartPre=/bin/mkdir --parents /etc/kubernetes/manifests
ExecStartPre=/bin/rm -f /var/lib/kubelet/cpu_manager_state
ExecStartPre=/bin/rm -f /var/lib/kubelet/memory_manager_state
EnvironmentFile=/etc/os-release
EnvironmentFile=-/etc/kubernetes/kubelet-workaround
EnvironmentFile=-/etc/kubernetes/kubelet-env
EnvironmentFile=/etc/node-sizing.env

ExecStart=/usr/local/bin/kubenswrapper \
    /usr/bin/kubelet \
      --config=/etc/kubernetes/kubelet.conf \
      --bootstrap-kubeconfig=/etc/kubernetes/kubeconfig \
      --kubeconfig=/var/lib/kubelet/kubeconfig \
      --container-runtime=remote \
      --container-runtime-endpoint=/var/run/crio/crio.sock \
      --runtime-cgroups=/system.slice/crio.service \
      --node-labels=node-role.kubernetes.io/control-plane,node-role.kubernetes.io/master,node.openshift.io/os_id=${ID} \
      --node-ip=${KUBELET_NODE_IP} \
      --address=${KUBELET_NODE_IP} \
      --minimum-container-ttl-duration=6m0s \
      --cloud-provider=external \
      --volume-plugin-dir=/etc/kubernetes/kubelet-plugins/volume/exec \
       \
      --hostname-override=${KUBELET_NODE_NAME} \
      --register-with-taints=node-role.kubernetes.io/master=:NoSchedule \
      --pod-infra-container-image=quay.io/openshift/okd-content@sha256:b28820e1027f7fe8e97430fcc7e91ea11b250f4b23524c6b2e252c81d2cc9903 \
      --system-reserved=cpu=${SYSTEM_RESERVED_CPU},memory=${SYSTEM_RESERVED_MEMORY},ephemeral-storage=${SYSTEM_RESERVED_ES} \
      --v=${KUBELET_LOG_LEVEL}

Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target``

I have fix to override kubelet.service file. I am not sure, if I am alone with this case.

Regards,

Thibaut

What exactly did you fix? We have some issues with kubelet too.

[tothlali]

upgrading to 4.12 the machine-config-daemon stucked:

rpm-ostree rebase --experimental ostree-unverified-registry:quay.io/openshift/okd-content@sha256:01d90a996a2e78a0486616a0adba0733db428f5a1074976054cedff62f17b2ac

Pulling manifest: ostree-unverified-image:docker://quay.io/openshift/okd-content@sha256:01d90a996a2e78a0486616a0adba0733db428f5a1074976054cedff62f17b2ac
Checking out tree 0a003a9... done
Inactive requests:
qemu-guest-agent (already provided by qemu-guest-agent-2:7.0.0-12.fc37.x86_64)
NetworkManager-ovs (already provided by NetworkManager-ovs-1:1.40.0-1.fc37.x86_64)
error: No enabled repositories

[tothlali]

Appendix: the airgapped system updated perfectly. The error mentioned above occurred at the online system

[mkaspar]

Tried yesterday from 4.11.0-0.okd-2023-01-14-152430 to 4.12.0-0.okd-2023-04-16-041331with severe failure. Need to explore yet but had to restore cluster from backup so the debugging is somewhat limited and if anybody had some hints...
We have cluster on vsphere environment started as OKD 4.3 or something like that so we've run into rpm-ostree rebase due tu glustrefs packages ( #1503 ) on master nodes (the only ones surviving from pre-4.10 era).

After removing the packages, the nodes restarted and failed to get to Ready state.
The reason was kubelet service not starting due to /run/resolv-prepender-kni-conf-done missing and the file was missing because the on-prem-resolv-prepender.service failed because of /run/resolv-prepender/env missing that should have been created by /etc/NetworkManager/dispatcher.d/30-resolv-prepender NM dispatcher script.

[mkaspar]

OK. Seems I've found the root cause and the solution so for anybody else running into this problem (not specific to 4.12) the root cause are SElinux labels on /etc/NetworkManager/dispatcher.d/ which prevent nm-dispatcher tu run /etc/NetworkManager/dispatcher.d/30-resolv-prepender script which in turn causes /run/resolv-prepender-kni-conf-done not to be created and kubectl.service to fail due to PreExec.
The problem seems to be related to the problem described in 1550 and so caused during some of the earlier updates where the rhcos-selinux-policy-upgrade.service service wasn't ran as the /usr/lib/os-release is missing RHEL_VERSION
In our case, relabeling /etc and running semodule -B seem to fix the issue.