Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Change error when draining a node with Pod Disruption Budget #797

Closed
OttaviaB opened this issue Nov 25, 2024 · 0 comments · Fixed by #798
Closed

Change error when draining a node with Pod Disruption Budget #797

OttaviaB opened this issue Nov 25, 2024 · 0 comments · Fixed by #798

Comments

@OttaviaB
Copy link
Contributor

SUMMARY

When k8s_drain tries to evict a pod which cannot be evicted due to a pod disruption budget an error "Too Many Requests" is returned.
Changing this message to something more meaningful would make it easier to work with retries/until loops in an Ansible playbook to drain the node.

ISSUE TYPE
  • Feature Idea
COMPONENT NAME

kubernetes.core.k8s_drain

ADDITIONAL INFORMATION

Current behaviour:

TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:37:09 +0100 (0:00:00.015)       0:00:00.015 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-c6s5w due to: Too Many Requests"}

Proposed behaviour:

TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
        kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
            ignore_daemonsets: true
            delete_emptydir_data: true
            wait_timeout: 100
            disable_eviction: false
            wait_sleep: 1
        delegate_to: localhost

See also:

#711

kubernetes/kubernetes#106286

softwarefactory-project-zuul bot pushed a commit that referenced this issue Dec 11, 2024
…798)

SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}


The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE


Feature Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <mgraves@redhat.com>
patchback bot pushed a commit that referenced this issue Dec 11, 2024
…798)

SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 *******
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}

The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE

Feature Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <mgraves@redhat.com>
(cherry picked from commit 52f2cb5)
patchback bot pushed a commit that referenced this issue Dec 11, 2024
…798)

SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 *******
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}

The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE

Feature Pull Request

COMPONENT NAME
k8s_drain

Reviewed-by: Mike Graves <mgraves@redhat.com>
(cherry picked from commit 52f2cb5)
softwarefactory-project-zuul bot pushed a commit that referenced this issue Dec 11, 2024
…798) (#815)

This is a backport of PR #798 as merged into main (52f2cb5).
SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}


The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE


Feature Pull Request

COMPONENT NAME
k8s_drain
softwarefactory-project-zuul bot pushed a commit that referenced this issue Dec 11, 2024
…798) (#816)

This is a backport of PR #798 as merged into main (52f2cb5).
SUMMARY
Closes #797 .
The error message "Too Many Requests" is confusing and is changed to a more meaningful message:
TASK [Drain node] *************************************************************************
Montag 25 November 2024  09:20:28 +0100 (0:00:00.014)       0:00:00.014 ******* 
fatal: [host -> localhost]: FAILED! => {"changed": false, "msg": "Failed to delete pod kube-public/draintest-6b84677b99-9jf7m due to: Cannot evict pod as it would violate the pod's disruption budget."}


The new task output would allow to deal with a pod disruption budget with the retries/until logic in a more controlled way:
---
- hosts: "{{ target }}"
  serial: 1
  gather_facts: false
  tasks:
    - name: Drain node
      kubernetes.core.k8s_drain:
        kubeconfig: "{{ kubeconfig_path }}"
        name: "{{ inventory_hostname }}"
        delete_options:
          ignore_daemonsets: true
          delete_emptydir_data: true
          wait_timeout: 100
          disable_eviction: false
          wait_sleep: 1
      delegate_to: localhost
      retries: 10
      delay: 5
      until: drain_result is success or 'disruption budget' not in drain_result.msg
      register: drain_result

ISSUE TYPE


Feature Pull Request

COMPONENT NAME
k8s_drain
softwarefactory-project-zuul bot pushed a commit that referenced this issue Jan 20, 2025
SUMMARY
Version 3.3.0 of ansible-collection kubernetes.core came with several improvements and bugfixes
ISSUE TYPE

New release pull request

Changelog
Minor Changes

k8s_drain - Improve error message for pod disruption budget when draining a node (#797).

Bugfixes

helm - Helm version checks did not support RC versions. They now accept any version tags. (#745).
helm_pull - Apply no_log=True to pass_credentials to silence false positive warning.. (#796).
k8s_drain - Fix k8s_drain does not wait for single pod (#769).
k8s_drain - Fix k8s_drain runs into a timeout when evicting a pod which is part of a stateful set  (#792).
kubeconfig option should not appear in module invocation log (#782).
kustomize - kustomize plugin fails with deprecation warnings (#639).
waiter - Fix waiting for daemonset when desired number of pods is 0. (#756).

ADDITIONAL INFORMATION
Collection kubernets.core version 3.3.0 is compatible with ansible-core>=2.14.0

Reviewed-by: Alina Buzachis
Reviewed-by: Yuriy Novostavskiy
Reviewed-by: Mike Graves <mgraves@redhat.com>
This was referenced Jan 20, 2025
softwarefactory-project-zuul bot pushed a commit that referenced this issue Jan 20, 2025
SUMMARY
This release came with new module helm_registry_auth, and improvements to the error messages in the k8s_drain module, new parameter insecure_registry for helm_template module and several bug fixes.
ISSUE TYPE

New release pull request

Changelog
Minor Changes

Bump version of ansible-lint to minimum 24.7.0 (#765).
Parameter insecure_registry added to helm_template as equivalent of insecure-skip-tls-verify (#805).
connection/kubectl.py - Added an example of using the kubectl connection plugin to the documentation (#741).
k8s_drain - Improve error message for pod disruption budget when draining a node (#797).

Bugfixes

helm - Helm version checks did not support RC versions. They now accept any version tags. (#745).
helm_pull - Apply no_log=True to pass_credentials to silence false positive warning.. (#796).
k8s_drain - Fix k8s_drain does not wait for single pod (#769).
k8s_drain - Fix k8s_drain runs into a timeout when evicting a pod which is part of a stateful set  (#792).
kubeconfig option should not appear in module invocation log (#782).
kustomize - kustomize plugin fails with deprecation warnings (#639).
waiter - Fix waiting for daemonset when desired number of pods is 0. (#756).

New Modules

helm_registry_auth - Helm registry authentication module

ADDITIONAL INFORMATION
Collection kubernets.core version 3.1.0 is compatible with ansible-core>=2.15.0

Reviewed-by: Mike Graves <mgraves@redhat.com>
yurnov added a commit to yurnov/kubernetes.core that referenced this issue Jan 20, 2025
SUMMARY
This release came with new module helm_registry_auth, and improvements to the error messages in the k8s_drain module, new parameter insecure_registry for helm_template module and several bug fixes.
ISSUE TYPE

New release pull request

Changelog
Minor Changes

Bump version of ansible-lint to minimum 24.7.0 (ansible-collections#765).
Parameter insecure_registry added to helm_template as equivalent of insecure-skip-tls-verify (ansible-collections#805).
connection/kubectl.py - Added an example of using the kubectl connection plugin to the documentation (ansible-collections#741).
k8s_drain - Improve error message for pod disruption budget when draining a node (ansible-collections#797).

Bugfixes

helm - Helm version checks did not support RC versions. They now accept any version tags. (ansible-collections#745).
helm_pull - Apply no_log=True to pass_credentials to silence false positive warning.. (ansible-collections#796).
k8s_drain - Fix k8s_drain does not wait for single pod (ansible-collections#769).
k8s_drain - Fix k8s_drain runs into a timeout when evicting a pod which is part of a stateful set  (ansible-collections#792).
kubeconfig option should not appear in module invocation log (ansible-collections#782).
kustomize - kustomize plugin fails with deprecation warnings (ansible-collections#639).
waiter - Fix waiting for daemonset when desired number of pods is 0. (ansible-collections#756).

New Modules

helm_registry_auth - Helm registry authentication module

ADDITIONAL INFORMATION
Collection kubernets.core version 3.1.0 is compatible with ansible-core>=2.15.0

Reviewed-by: Mike Graves <mgraves@redhat.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging a pull request may close this issue.

1 participant