[Release-1.24] Segmentation Fault in Upgrade Jobs - Outdated Helm-Klipper RKE2

[rancherbot]

This is a backport issue for #4641, automatically created via rancherbot by @brandond

Original issue description:

Environmental Info:
RKE2 Version:

1.27.4+rke2r1
Node(s) CPU architecture, OS, and Version:

Any cluster config.
Cluster Configuration:

Any cluster config.
Describe the bug:

Steps To Reproduce:

• Installed RKE2:

Expected behavior:

Actual behavior:

Additional context / logs:

Reference k3s issue with details: k3s-io/k3s#8208
Reference helm update issue: helm/helm#12061
Reference helm-klipper pull request: k3s-io/klipper-helm#68

[brandond]

I don't believe the specific issue that triggers the helm crash occurs when using Kubernetes v1.24, but it's good to get the version bumped anyways. Should just verify that the new klipper-helm image is used and works normally.

[VestigeJ]

##Environment Details
VERSION=v1.24.16+rke2r1
VERSION=v1.24.17-rc1+rke2r1

Infrastructure

• Cloud

Node(s) CPU architecture, OS, and version:

ami-0bbc06589f2e4f4f2

Linux 5.14.21-150500.53-default x86_64 GNU/Linux

PRETTY_NAME="SUSE Linux Enterprise Server 15 SP5"

Cluster Configuration:

NAME               STATUS   ROLES                       AGE     VERSION
ip-12-13-12-15     Ready    control-plane,etcd,master   5m35s   v1.24.16+rke2r1 

Config.yaml:

write-kubeconfig-mode: 644
debug: true
token: YOUR_TOKEN_HERE
profile: cis-1.6
selinux: true

Reproduction

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_RKE2_VERSION=v1.24.16+rke2r1 INSTALL_RKE2_EXEC=server ./install-rke2.sh 
$ go_rke2 // sudo systemctl enable rke2-server --now
$ set_kubefig // export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
$ kgn // kubectl get nodes 
$ get_images | grep -i klipper

Results:

$ get_images | grep -i klipper

docker.io/rancher/klipper-helm                                  v0.8.0-build20230510                       6f42df210d7fa       95MB

Validation

$ curl https://get.rke2.io --output install-"rke2".sh
$ sudo chmod +x install-"rke2".sh
$ sudo groupadd --system etcd && sudo useradd -s /sbin/nologin --system -g etcd etcd
$ sudo modprobe ip_vs_rr
$ sudo modprobe ip_vs_wrr
$ sudo modprobe ip_vs_sh
$ sudo printf "on_oovm.panic_on_oom=0 \nvm.overcommit_memory=1 \nkernel.panic=10 \nkernel.panic_ps=1 \nkernel.panic_on_oops=1 \n" > ~/60-rke2-cis.conf
$ sudo cp 60-rke2-cis.conf /etc/sysctl.d/
$ sudo systemctl restart systemd-sysctl
$ sudo INSTALL_RKE2_VERSION=v1.24.17-rc1+rke2r1 INSTALL_RKE2_EXEC=server ./install-rke2.sh 
$ go_rke2 // sudo systemctl enable rke2-server --now
$ set_kubefig // export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
$ kgn // kubectl get nodes 
$ get_images | grep -i klipper

Results:

$ get_images | grep -i klipper

docker.io/rancher/klipper-helm                                  v0.8.2-build20230815                       5f89cb8137ccb       90.9MB

Additional context / logs:

$ get_images() {
    _product="${1:-$PRODUCT}"
    case "${_product}" in
    rke2) sudo /var/lib/rancher/"${_product}"/bin/crictl --config /var/lib/rancher/"${_product}"/agent/etc/crictl.yaml ps
          sudo /var/lib/rancher/"${_product}"/bin/crictl --config /var/lib/rancher/"${_product}"/agent/etc/crictl.yaml images
        ;;
    k3s) 
        sudo k3s crictl ps
        sudo k3s crictl img ls
        ;;
    esac
}