Skip to content

NodeSelectors and tolerations

Jump to bottom
Sherin Varughese edited this page Jul 17, 2024 · 6 revisions

Taint and Toleration Scenarios for Search Pods

Overview

Taints and tolerations in Kubernetes are mechanisms to ensure that pods are scheduled only on appropriate nodes. A taint on a node allows the node to repel a set of pods. Tolerations are applied to pods to allow them to schedule onto nodes with matching taints. This document covers how search pods respond under different taint/toleration configurations on the hub and managed clusters.

On the Hub Cluster

1. Default Configuration

Search Pods: On the hub cluster, pods have infra tolerations. No node selectors specified on the hub pods.

Expected Behavior:

Hub cluster pods will have infra tolerations by default, so that pods can get scheduled on infra tainted nodes, if available. These infra tolerations are propagated by the multiclusterhub-operator pod to the Search CR (search-v2-operator) and passed on to Search pods on the hub. https://github.com/stolostron/multiclusterhub-operator/blob/b3fef20c912509eb7fdc3593e95fac1859ae8603/pkg/utils/utils.go#L481

Search Pods on Hub: The search pods will have infra tolerations passed down by the mch-operator. The pods can run on any node. If there are any nodes with infra label and taints, then the pods can be scheduled on these nodes.
Outcome: Search functionality should work seamlessly on the hub, and the pods should get scheduled on any node.

2. Customize MCH with a Node Selector

Hub Cluster: Infra node selector added on MCH.

Expected Behavior: Search Pods on Hub: This infra node selector should get propagated to Search CR and then to the Search pods. Search pods would update and restart with the added node selector. If there are no matching nodes available, pods will remain pending. If there is a matching node with infra label and taints, the new pods will get scheduled there.
Outcome: Search functionality should work seamlessly on the hub and the pods should get scheduled on the node with infra label and taints. No changes on managed cluster pods.

How to test:

  1. Add nodeSelector to MCH:
    spec:
    nodeSelector:
    node-role.kubernetes.io/infra: "

  2. Check Search CR to ensure the infra nodeSelector is propagated there.

  3. Check if Search pods are restarted with the new nodeSelector.

  4. If a matching node with infra label and taints is available, the new pods should get scheduled there. Else, they will remain pending.

  5. If there are no matching nodes, add infra label and taint to a node on hub cluster.
    kubectl label nodes <node-name> node-role.kubernetes.io/infra=""
    kubectl taint nodes <node-name> node-role.kubernetes.io/infra=:NoSchedule

  6. Check if all Search pods are running on the new node.

On the Managed Cluster

1. Default Configuration

Search Pods: On the managed cluster, no tolerations or node selectors are specified for the addon pods.

Expected Behavior:

Search Pods on the managed cluster: The search pods can run on any node.
Outcome: Search functionality should work seamlessly on the managed cluster, and the pods should get scheduled on any node.

2. Customize with a Node Selector

Managed Cluster: Infra node selector added to the pods via AddonDeploymentConfig.

Expected Behavior:
Search Pods on the Managed Cluster: This infra node selector and tolerations should get propagated from the AddonDeploymentConfig and ClusterManagementAddon to the Search pods. Search pods would update and restart with the added node selector and tolerations. If there are no matching nodes available, pods will remain pending. If there is a matching node with infra label and taints, the new pods will get scheduled there.
Outcome: Search functionality should work seamlessly on the managed cluster and the pods should get scheduled on the node with infra label and taints. No changes on hub cluster pods.

How to test:

  1. Create Search AddonDeploymentConfig on the managed cluster:
apiVersion: addon.open-cluster-management.io/v1alpha1
kind: AddOnDeploymentConfig
metadata:
  name: search-collector
  namespace: open-cluster-management
spec:
  nodePlacement:
    nodeSelector:
      node-role.kubernetes.io/infra: ""
    tolerations:
    - effect: NoSchedule
      key: node-role.kubernetes.io/infra
      operator: Exists

oc apply -f addondeploymentconfig

  1. Use the configuration that you created as the global default configuration for your add-on by running the following command:
oc patch clustermanagementaddons search-collector --type='json' -p='[{"op":"add", "path":"/spec/supportedConfigs", "value":[{"group":"addon.open-cluster-management.io","resource":"addondeploymentconfigs", "defaultConfig":{"name":"search-collector","namespace":"open-cluster-management"}}]}]'
  1. Check if Addon Search pods are restarted with the new nodeSelector and tolerations.
  2. If a matching node with infra label and taints is available, the new pods should get scheduled there. Else, they will remain pending.
  3. If there are no matching nodes, add infra label and taint to a node on the managed cluster.
    kubectl label nodes <node-name> node-role.kubernetes.io/infra="" kubectl taint nodes <node-name> node-role.kubernetes.io/infra=:NoSchedule
  4. Check if the klusterlet-addon-search-xxxxx pod is running on the new node.

Additional information in https://docs.redhat.com/en/documentation/red_hat_advanced_cluster_management_for_kubernetes/2.10/html/add-ons/add-ons-overview#add-ons-klusterlet

Clone this wiki locally