k8s_drain runs into a timeout when evicting a pod which is part of a stateful set

[OttaviaB]

SUMMARY

k8s_drain runs into a timeout when evicting a pod which is part of a stateful set.

The pod gets the same name on a different node and because k8s_drain checks only the pod name, but not the node name, it thinks that the original pod is still running.

ISSUE TYPE

• Bug Report

COMPONENT NAME

k8s_drain

ANSIBLE VERSION

ansible [core 2.15.12]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/venvs/ansible/lib64/python3.9/site-packages/ansible
  ansible collection location = ~/.ansible/collections:/usr/share/ansible/collections
  executable location = ~/venvs/ansible/bin/ansible
  python version = 3.9.19 (main, Mar 22 2024, 21:01:47) [GCC] (~/venvs/ansible/bin/python3.9)
  jinja version = 3.1.4
  libyaml = True

COLLECTION VERSION

# ~/.ansible/collections/ansible_collections
Collection                    Version
----------------------------- -------
kubernetes.core               5.0.0

CONFIGURATION

CALLBACKS_ENABLED(~/.ansible.cfg) = ['ansible.posix.profile_tasks']
CONFIG_FILE() = ~/.ansible.cfg
PAGER(env: PAGER) = less

OS / ENVIRONMENT

• Ansible controller: SLES 15-SP5
• Remote hosts: SLES 15-SP5
• RKE2 Version: v1.28.10+rke2r1

STEPS TO REPRODUCE

• Deploy a stateful set
• Drain the node where the stateful set's pod is running

---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  vars:
    kubeconfig_dir: "/home/ansible/.kubeconfigs"
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          force: true
          wait_sleep: 1
      delegate_to: localhost

EXPECTED RESULTS

k8s_drain should return directly after the pods are evicted.

ACTUAL RESULTS

k8s_drain keeps going until the timeout is reached although the pods are long gone. It then returns a warning.

PLAY [Test] *******************************************************************

TASK [Drain node] *****************************************************************
Donnerstag 18 Juli 2024  14:40:13 +0200 (0:00:00.051)       0:00:00.051 ******* 
[WARNING]: cannot delete mirror Pods using API server: kube-system/kube-proxy-<node_name>.
[WARNING]: timeout reached while pods were still running.
changed: [<node_name> -> localhost]

PLAY RECAP ************************************************************************
<node_name>             : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Donnerstag 18 Juli 2024  14:41:55 +0200 (0:01:41.902)       0:01:41.953 ******* 
=============================================================================== 
Drain node --------------------------------------------------------------- 101.90s

This happens because the function wait_for_pod_deletion in k8s_drain never checks on which node a pod is actually running:

            try:
                response = self._api_instance.read_namespaced_pod(
                    namespace=pod[0], name=pod[1]
                )
                if not response:
                    pod = None
                time.sleep(wait_sleep)

The conditon if not response is never met, because the new pod has the same name as the old one.