k8s_drain not waiting for pod deletion if there is only one pod to evict

[OttaviaB]

SUMMARY

In the module k8s_drain the result of the draining process is never checked if there is only one pod to evict from the node.

ISSUE TYPE

• Bug Report

COMPONENT NAME

k8s_drain

ANSIBLE VERSION

ansible [core 2.15.12]
  config file = /etc/ansible/ansible.cfg
  configured module search path = ['~/.ansible/plugins/modules', '/usr/share/ansible/plugins/modules']
  ansible python module location = ~/venvs/ansible/lib64/python3.9/site-packages/ansible
  ansible collection location = ~/.ansible/collections:/usr/share/ansible/collections
  executable location = ~/venvs/ansible/bin/ansible
  python version = 3.9.19 (main, Mar 22 2024, 21:01:47) [GCC] (~/venvs/ansible/bin/python3.9)
  jinja version = 3.1.4
  libyaml = True

COLLECTION VERSION

# ~/.ansible/collections/ansible_collections
Collection                    Version
----------------------------- -------
kubernetes.core               5.0.0

CONFIGURATION

CALLBACKS_ENABLED(~/.ansible.cfg) = ['ansible.posix.profile_tasks']
CONFIG_FILE() = ~/.ansible.cfg
PAGER(env: PAGER) = less

OS / ENVIRONMENT

• Ansible controller: SLES 15-SP5
• Remote hosts: SLES 15-SP5
• RKE2 Version: v1.28.10+rke2r1

STEPS TO REPRODUCE

• deploy a single pod running only on one node
  • Problem is to be observed best with a pod that takes a long time to terminate.
  • e. g. a deployment like this:

apiVersion: apps/v1
kind: Deployment
metadata:
  name: draintest
  namespace: kube-public
spec:
  progressDeadlineSeconds: 600
  replicas: 1
  revisionHistoryLimit: 10
  selector:
    matchLabels:
      app: draintaints
  strategy:
    rollingUpdate:
      maxSurge: 25%
      maxUnavailable: 25%
    type: RollingUpdate
  template:
    metadata:
      labels:
        app: draintaints
    spec:
      affinity:
        nodeAffinity:
          preferredDuringSchedulingIgnoredDuringExecution:
            - preference:
                matchExpressions:
                  - key: kubernetes.io/hostname
                    operator: In
                    values:
                      - <node_name>
              weight: 100
      containers:
        - command:
            - bash
            - '-c'
            - |
              trap 'echo -e "\nShutting down..."; sleep 60; exit' SIGTERM
              while true; do 
                  echo "Hi, I'm Steve!";
                  sleep 1
              done
          image: <registry>/shell:v2.0
          imagePullPolicy: IfNotPresent
          name: main
          resources: {}
          securityContext:
            allowPrivilegeEscalation: false
            capabilities:
              drop:
                - ALL
            privileged: false
            readOnlyRootFilesystem: true
            runAsGroup: 12345
            runAsNonRoot: true
            runAsUser: 12345
            seccompProfile:
              type: RuntimeDefault
          terminationMessagePath: /dev/termination-log
          terminationMessagePolicy: File
      dnsPolicy: ClusterFirst
      restartPolicy: Always
      schedulerName: default-scheduler
      securityContext: {}
      terminationGracePeriodSeconds: 200

• drain this node

---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  vars:
    kubeconfig_dir: "~/.kubeconfigs"
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          terminate_grace_period: 30
          force: true
      delegate_to: localhost

EXPECTED RESULTS

k8s_drain should wait until the pod is evicted successfully or run in a timeout.

ACTUAL RESULTS

k8s_drain reports immediately status "changed" no matter if the pod was evicted or not.

PLAY [Test] ****************************************************************************************************************************************************************************

TASK [Drain node] **************************************************************************************************************************************************************************
Montag 15 Juli 2024  15:48:57 +0200 (0:00:00.047)       0:00:00.047 *********** 
[WARNING]: cannot delete mirror Pods using API server: kube-system/kube-proxy-<node_name>.
changed: [<node_name> -> localhost]

PLAY RECAP *********************************************************************************************************************************************************************************
<node_name>             : ok=1    changed=1    unreachable=0    failed=0    skipped=0    rescued=0    ignored=0   

Montag 15 Juli 2024  15:48:58 +0200 (0:00:01.202)       0:00:01.249 *********** 
=============================================================================== 
Drain node -------------------------------------------------------------------------------------------------------------------------------------------------------------------------- 1.20s

Even though the pod needs 60 seconds to terminate, k8s_drain reports deletion already after 2 seconds while the pod is still running.