Upgrade 1.27.12 to 1.28.8.

[xhejtman]

Hello,

I encountered issue in the following situation. I did upgrade rke2 from 1.27.10 to 1.27.12, but no reboot, just rke2-[server/agent] restart. Then, still no reboot, I did upgrade from 1.27.12 to 1.28.8. It was mostly ok, however, in about two cases (of 25 nodes) I run into the issue that containerd did not start. The process was running, but connection on the socket was refused.

Multiple rke2 restarts do not help. Even reverting back to 1.27.12 version and restart does not help. Node reboot solves it.

Is it something known/expected?

[brandond]

Have you looked at the containerd.log file to see why it's not starting?

[xhejtman]

containerd.log

[brandond]

It appears to be taking a long time to remove stale containers from a prior run. I've not seen this before, but it appears to be pausing for 5 seconds between each container, which will take a while if there are many old containers to clean up.

Have you tried running the rke2-killall.sh script?

[xhejtman]

Have you tried running the rke2-killall.sh script?

no, but it is probably the same as reboot and should help. I just wanted to avoid kill/restart node. Not able to reproduce, but can test next time.

Btw, can I remove the stale containers in advance?

[brandond]

If there is anything left after running the killall script, you should be able to remove everything under /run/k3s/containerd/

[xhejtman]

thank you.