Skip to content

Latest commit

 

History

History
52 lines (37 loc) · 3.37 KB

tolerations.md

File metadata and controls

52 lines (37 loc) · 3.37 KB
Raw

Taints and Tolerations for Seeds and Shoots

Similar to taints and tolerations for Nodes and Pods in Kubernetes, the Seed resource supports specifying taints (.spec.taints, see this example) while the Shoot resource supports specifying tolerations (.spec.tolerations, see this example). The feature is used to control scheduling to seeds as well as decisions whether a shoot can use a certain seed.

Compared to Kubernetes, Gardener's taints and tolerations are very much down-stripped right now and have some behavioral differences. Please read the following explanations carefully if you plan to use them.

Scheduling

When scheduling a new shoot, the gardener-scheduler will filter all seed candidates whose taints are not tolerated by the shoot. As Gardener's taints/tolerations don't support effects yet, you can compare this behaviour with using a NoSchedule effect taint in Kubernetes.

Be reminded that taints/tolerations are no means to define any affinity or selection for seeds - please use .spec.seedSelector in the Shoot to state such desires.

⚠️ Please note that - unlike how it's implemented in Kubernetes - a certain seed cluster may only be used when the shoot tolerates all the seed's taints. This means that specifying .spec.seedName for a seed whose taints are not tolerated will make the gardener-apiserver reject the request.

Consequently, the taints/tolerations feature can be used as means to restrict usage of certain seeds.

Toleration Defaults and Whitelist

The Project resource features a .spec.tolerations object that may carry defaults and a whitelist (see this example). The corresponding ShootTolerationRestriction admission plugin (cf. Kubernetes' PodTolerationRestriction admission plugin) is responsible for evaluating these settings during creation/update of Shoots.

Whitelist

If a shoot gets created or updated with tolerations, then it is validated that only those tolerations may be used that were added to either a) the Project's .spec.tolerations.whitelist, or b) to the global whitelist in the ShootTolerationRestriction's admission config (see this example).

⚠️ Please note that the tolerations whitelist of Projects can only be changed if the user trying to change it is bound to the modify-spec-tolerations-whitelist custom RBAC role, e.g., via the following ClusterRole:

apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: full-project-modification-access
rules:
- apiGroups:
  - core.gardener.cloud
  resources:
  - projects
  verbs:
  - create
  - patch
  - update
  - modify-spec-tolerations-whitelist
  - delete

Defaults

If a shoot gets created, then the default tolerations specified in both the Project's .spec.tolerations.defaults and the global default list in the ShootTolerationRestriction admission plugin's configuration will be added to the .spec.tolerations of the Shoot (unless it already specifies a certain key).