pause version mismatch on extra node after kubeadm join

[dv-test-gh]

What keywords did you search in kubeadm issues before filing this one?

pause version problem join

Is this a BUG REPORT or FEATURE REQUEST?

BUG REPORT

Versions

kubeadm version (use kubeadm version):

kubeadm version: &version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:24:38Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}

Environment:

• Kubernetes version (use kubectl version):
  Client Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.1", GitCommit:"3ddd0f45aa91e2f30c70734b175631bec5b5825a", GitTreeState:"clean", BuildDate:"2022-05-24T12:26:19Z", GoVersion:"go1.18.2", Compiler:"gc", Platform:"linux/amd64"}
  Kustomize Version: v4.5.4
  Server Version: version.Info{Major:"1", Minor:"24", GitVersion:"v1.24.3", GitCommit:"aef86a93758dc3cb2c658dd9657ab4ad4afc21cb", GitTreeState:"clean", BuildDate:"2022-07-13T14:23:26Z", GoVersion:"go1.18.3", Compiler:"gc", Platform:"linux/amd64"}

• Cloud provider or hardware configuration:

• OS (e.g. from /etc/os-release):
  NAME="CentOS Stream"
  VERSION="8"
  ID="centos"
  ID_LIKE="rhel fedora"
  VERSION_ID="8"
  PLATFORM_ID="platform:el8"
  PRETTY_NAME="CentOS Stream 8"
  ANSI_COLOR="0;31"
  CPE_NAME="cpe:/o:centos:centos:8"
  HOME_URL="https://centos.org/"
  BUG_REPORT_URL="https://bugzilla.redhat.com/"
  REDHAT_SUPPORT_PRODUCT="Red Hat Enterprise Linux 8"
  REDHAT_SUPPORT_PRODUCT_VERSION="CentOS Stream"

• Kernel (e.g. uname -a):
  Linux XXX 4.18.0-394.el8.x86_64 kubeadm join on slave node fails preflight checks #1 SMP Tue May 31 16:19:11 UTC 2022 x86_64 x86_64 x86_64 GNU/Linux

• Container runtime (CRI) (e.g. containerd, cri-o):
  cri-o-1.21.3-1.module_el8+12969+ebff6d46.x86_64

• Container networking plugin (CNI) (e.g. Calico, Cilium):
  calico v3.23

• Others:

What happened?

Problem:
After joining (kubeadm join) a new node (with no pre-installed images), I get only pause:3.5.
However, the /var/lib/kubelet/kubeadm-flags.env file contains the same as on the main node, that is:
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --pod-infra-container-image=k8s.gcr.io/pause:3.7"

In /var/log/messages, on the new node, we get:
Jul 10 03:28:48 localhost crio[1248]: time="2022-07-10 03:28:48.288405920-04:00" level=info msg="Checking image status: k8s.gcr.io/pause:3.7" id=c5f090a6-0c1f-4221-9bcb-716b879ca4cb name=/runtime.v1.ImageService/ImageStatus
Jul 10 03:28:48 localhost crio[1248]: time="2022-07-10 03:28:48.288573150-04:00" level=info msg="Image k8s.gcr.io/pause:3.7 not found" id=c5f090a6-0c1f-4221-9bcb-716b879ca4cb name=/runtime.v1.ImageService/ImageStatus

A workaround is to do "podman pull k8s.gcr.io/pause:3.7" directly on the new node.

What you expected to happen?

Correct versions of kubernetes infrastructure be installed without manual intervention.

How to reproduce it (as minimally and precisely as possible)?

Starting system from scratch, with no pre-installed images (podman image rm -a).
On main node (kubeadm init), I get two versions of the pause container: 3.5 and 3.7.
The reference to version 3.7 seems to come from /var/lib/kubelet/kubeadm-flags.env:
KUBELET_KUBEADM_ARGS="--container-runtime=remote --container-runtime-endpoint=unix:///var/run/crio/crio.sock --pod-infra-container-image=k8s.gcr.io/pause:3.7"
The reference to version 3.5 seems to come from cri-o, crio-status config yields:
[crio.image]
pause_image = "k8s.gcr.io/pause:3.5"

(Are these references to two different versions a problem?)

Now, in /var/log/messages, on the main node, we get:
Jul 16 16:45:19 localhost crio[1311004]: time="2022-07-16 16:45:19.570935505-04:00" level=info msg="Checking image status: k8s.gcr.io/pause:3.7" id=8d3456d7-6a7a-4cb1-a1f8-df9b61e8e9bb name=/runtime.v1.ImageService/ImageStatus
Jul 16 16:45:19 localhost crio[1311004]: time="2022-07-16 16:45:19.571631830-04:00" level=info msg="Image status: &{0xc00034a070 map[]}" id=8d3456d7-6a7a-4cb1-a1f8-df9b61e8e9bb name=/runtime.v1.ImageService/ImageStatus

And after joining a new node, on the new node, we get a "not found" error in /var/log/messages about pause:3.7.

Anything else we need to know?

[neolit123]

The runtime page explains how to explicitly configure the pause image for crio and containerd in their respective config.

https://kubernetes.io/docs/setup/production-environment/container-runtimes/

The pod-infra-image flag at this point only tells the kubelet to not gc this image.

[neolit123]

/kind support

[dv-test-gh]

Thanks for pointing to this config of crio.
I've therefore done kubeadm config images list to get the correct version of the pause container (since the reference from the page above listed 3.6 and my setup seems to want 3.7).
I've added this config to crio.conf, sent HUP to crio, checked with systemctl status crio.
Then started a fresh install with kubeadm init and kubeadm join and now I only get pause:3.7.
No more "not found" messages in /var/log/messages.
Thanks.