[container-engine/runc : runc | set is_ostree] failing during cluster upgrade to 1.24.1 on Ubuntu 20.04

[Vloupmon]

Environment:

• Cloud provider or hardware configuration:
  Bare-metal

• OS (printf "$(uname -srm)\n$(cat /etc/os-release)\n"):
  Linux 5.4.0-122-generic x86_64
  NAME="Ubuntu"
  VERSION="20.04.4 LTS (Focal Fossa)"
  ID=ubuntu
  ID_LIKE=debian
  PRETTY_NAME="Ubuntu 20.04.4 LTS"
  VERSION_ID="20.04"
  HOME_URL="https://www.ubuntu.com/"
  SUPPORT_URL="https://help.ubuntu.com/"
  BUG_REPORT_URL="https://bugs.launchpad.net/ubuntu/"
  PRIVACY_POLICY_URL="https://www.ubuntu.com/legal/terms-and-policies/privacy-policy"
  VERSION_CODENAME=focal
  UBUNTU_CODENAME=focal

• Version of Ansible (ansible --version):
  2.12.5

• Version of Python (python --version):
  3.8.10

Kubespray version (commit) (git rev-parse --short HEAD):
a270632

Network plugin used:
Calico

Full inventory with variables (ansible -i inventory/sample/inventory.ini all -m debug -a "var=hostvars[inventory_hostname]"):

https://gist.github.com/Vloupmon/d92dfa53718d0725d2c7f24ce7225291

Command used to invoke ansible:

ANSIBLE_STDOUT_CALLBACK=yaml ansible-playbook -vv \
    -i ${DIR}/var/${CLUSTER}/inventory/inventory.ini \
    --private-key=${DIR}/var/${CLUSTER}/key \
    --become --become-user=root \
    -e generic_user=${GENERIC_USER} \
    -e ansible_user=${ANSIBLE_USER} \
    -e kube_version="v${KUBE_VERSION}"  \
    ${KUBESPRAY}/upgrade-cluster.yml

Output of ansible run:

The meaningful output is this part :

TASK [container-engine/runc : runc | set is_ostree] 
****************************************************************************************************
task path: /home/ubuntu/k8s/kubespray-cluster-maintenance/kubespray/roles/container-engine/runc/tasks/main.yml:2
fatal: [k8s-control-plane-1]: FAILED! =>
  msg: |-
    The task includes an option with an undefined variable. The error was: 'dict object' has no attribute 'stat'

    The error appears to be in '/home/ubuntu/k8s/kubespray-cluster-maintenance/kubespray/roles/container-engine/runc/tasks/main.yml': line 2, column 3, but may
    be elsewhere in the file depending on the exact syntax problem.

    The offending line appears to be:

    ---
    - name: runc | set is_ostree
      ^ here

Anything else do we need to know:

I've been planning the upgrade of a few of our clusters running older versions of k8s and deployed by the previous team with an old version of Kubespray by going through the entire sequential upgrade process on a few test machines.
I had to resolve a few bugs and integrate changes along the way but it's been altogether going well.
But now trying to upgrade a 1.23.5 cluster to 1.24.1 I'm not getting stuck on the docker->containerd transition but what looks like an error related to a rather trivial distro thing.

[ghost]

i have same issue, anyone knows how resolve this.

[shelmingsong]

the same issue on CentOS 7.5.1804

[shelmingsong]

I reset the cluster with reset.yaml and then use cluster.yaml to create the cluster so that the operation works properly.

Note: executing reset.yaml will clear the cluster, please do so after careful consideration.

[electrocucaracha]

I've submitted a fix for this issue, hopefully this gets merged soon and available in the next release.