Upgrading Cluster from 1.23.9 to 1.24.10 not possible

[michaelschefczyk]

Versions

kubeadm version: 1.23.9

Environment:
Kubernetes version: 1.23.9
Cloud provider or hardware configuration: Self-hosted virtual machine
OS: Debian GNU/Linux 11 (bullseye)
Kernel: 5.10.0-21-amd64 #1 SMP Debian 5.10.162-1 (2023-01-21) x86_64 GNU/Linux
Container runtime: containerd github.com/containerd/containerd 1.4.13ds1 1.4.13ds1-1~deb11u3
Container networking plugin (CNI): rancher/mirrored-flannelcni-flannel v0.19.1

What happened?

After six months, I did intend to upgrade the cluster from 1.23.9 to 1.24.10
based on: https://kubernetes.io/docs/tasks/administer-cluster/kubeadm/kubeadm-upgrade/

Upon kubeadm upgrade apply v1.24.10, Errors:

[preflight] Some fatal errors occurred:
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-apiserver:v1.24.10: output: time="2023-02-03T18:17:55+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-controller-manager:v1.24.10: output: time="2023-02-03T18:17:55+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-scheduler:v1.24.10: output: time="2023-02-03T18:17:55+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/kube-proxy:v1.24.10: output: time="2023-02-03T18:17:55+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/pause:3.7: output: time="2023-02-03T18:17:56+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/etcd:3.5.6-0: output: time="2023-02-03T18:17:56+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1
[ERROR ImagePull]: failed to pull image registry.k8s.io/coredns/coredns:v1.8.6: output: time="2023-02-03T18:17:56+01:00" level=fatal msg="validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock": rpc error: code = Unimplemented desc = unknown service runtime.v1.ImageService"
, error: exit status 1

editing kubectl -n kube-system edit cm kubeadm-config
imageRepository back to k8s.gcr.io does not work -> overwritten by registry.k8s.io each time

updating containerd to newer version from docker repo does not work, because then the API server is no longer available.

P.S. Support question: Is there a way out of this or do I kill my cluster and start over from scratch?

What you expected to happen?

Clean upgrade, as before.

How to reproduce it (as minimally and precisely as possible)?

Uncertain. It does happen every time I try it from a reverted version.

Anything else we need to know?

[kundan2707]

/retitle Upgrading Cluster from 1.23.9 to 1.24.10 not possible

[SataQiu]

Hi @michaelschefczyk
Thanks for your feedback.

validate service connection: CRI v1 image API is not implemented for endpoint "unix:///var/run/cri-dockerd.sock"

It appears that you are still using docker as the container runtime, and it does not support the runtime.v1 api. What's the cri-dockerd version?
And maybe you can try migrating docker to containerd with https://kubernetes.io/docs/tasks/administer-cluster/migrating-from-dockershim/change-runtime-containerd/

[neolit123]

/support

[github-actions]

Hello, @michaelschefczyk 🤖 👋

You seem to have troubles using Kubernetes and kubeadm.
Note that our issue trackers should not be used for providing support to users.
There are special channels for that purpose.

Please see:

• https://github.com/kubernetes/kubeadm#support

[neolit123]

yes, if using docker try moving to containerd
it's manual

[michaelschefczyk]

No need to worry about me, but please consider working on more telling error messages!

I did resolve my case by seeting up a new cluster. Of course, this is a matrix of n-dimensional complexity: Debian bullseye does not provide a containerd-package compatible with Kubernetes > 1.25, so one needs to move to another source (in my case docker repository). Then, if one wants to use a private repository (outside docker, so that is is available even when the cluster is off), one does need tls where the certificates are trusted and providing the name via SAN (unclear if this is imposed by the particular containerd-version or by kubelet - there is documentation to skip certificate verification but it does not seem to match the latest version of containerd and/or kubelet). Hence, I made my respository available via a reverse proxy (haproxy).

[karlvr]

I have just experienced this problem and I believe the image pull errors that you experienced @michaelschefczyk are because your installed version of cri-tools was newer than your Kubernetes binaries. I didn't control the version of cri-tools and because kubeadm depends on >= I had the latest installed, which was 1.26, and which is incompatible with the older version of cri-dockerd because of https://kubernetes.io/blog/2022/12/09/kubernetes-v1-26-release/#cri-v1alpha2-removed

You can remedy the issue by downgrading cri-tools:

apt install cri-tools=1.23.17-00

Now that I've upgraded to cri-dockerd 0.3.2 my cri-tools works and I can upgrade.

[neolit123]

i am surprised cri-dockerd is as maintained. note that the original estimate was that the project will not be able to keep up.

in any case, k8s recommends containerd or crio.
containerd in particular is recommended to be installed from its github readme / packages and not from a distro:
https://kubernetes.io/docs/setup/production-environment/container-runtimes/