Configure fine-granular PriorityClasses for all components

[vpnachev]

How to categorize this issue?

/area high-availability ops-productivity quality
/kind enhancement

What would you like to be added:

Configure priorityclass for the Gardener components

• admission-controller -> Add PriorityClass to gardener control plane components #5652
• apiserver -> Add PriorityClass to gardener control plane components #5652
• controller-manager -> Add PriorityClass to gardener control plane components #5652
• gardenlet -> Deploy gardenlet with highest possible priority #1899
• scheduler -> Add PriorityClass to gardener control plane components #5652
• seed-admission-controller -> Add PriorityClass to gardener-seed-admission-controller and gardener-resource-manager #5657
• resource-manager -> Add PriorityClass to gardener-seed-admission-controller and gardener-resource-manager #5657

With the new plan from #5634 (comment):

• Introduce well-known and fine-granular PriorityClasses managed by gardenlet #6186
• adapt all components managed by gardenlet to use the well-known PriorityClasses and drop custom ones
  • gardenlet itself (helm chart, reused by ManagedSeed) -> Adapt gardenlet to use priority class gardener-system-critical #6235
    • Revert changes Revert "Adapt gardenlet to use priority class gardener-system-critical (#6235)"  #6432
    • Remove gardenlet priority class and use gardener-system-critical-migration priority class for migration purpose of gardenlet priority class. -> Migrate gardenlet priority-class #6510
    • Use gardener-system-critical for gardenlet. -> Use priority class gardener-system-critical #6586
    • Remove clean up code of gardener-system-critical-migration priority class. -> Cleanup old priority class and related migration code  #6738
  • Seed system components: seed-admission-controller, resource-manager, istiod, etc. -> Adapt seed system components priority classes #6257
  • Shoot control plane components
    • kube-apiserver, kube-controller-manager -> Adapt the kube-apiserver and kube-controller-manager to use the appropriate PriorityClasses #6242
  • etcd: this will cause a rollout of all etcd pods, should be done separately and preferably together with another change that rolls out etcd: Integrate etcd-druid v0.12 #6467
• adapt all extensions components deployed to seeds (included in ControllerDeployment) and control plane / shoot system components managed by extensions to use the well-known PriorityClasses and drop custom ones
  • provider-local -> Adapt provider-local to use priority class gardener-system-900 #6236
  • external-dns-management -> Adapt the priority accodring to the gardener guideline external-dns-management#265
  • kupid -> Adapt the deployment priorityClassName according to the gardener documentation kupid#44
  • networking-calico -> Adapt the deployment priorityClassName according to the gardener documentation  gardener-extension-networking-calico#192
  • networking-cilium - > Adapt the deployment priorityClassName according to the gardener documentation gardener-extension-networking-cilium#103
  • os-coreos -> Adapt the deployment priorityClassName according to the gardener documentation gardener-extension-os-coreos#46
  • os-gardenlinux -> Adapt the deployment priorityClassName according to the gardener documentation gardener-extension-os-gardenlinux#64
  • os-suse-chost -> Adapt the deployment priorityClassName according to the gardener documentation gardener-extension-os-suse-chost#69
  • os-ubuntu -> Adapt the deployment priorityClassName according to the gardener documentation gardener-extension-os-ubuntu#66
  • provider-alicloud -> Adapt the priority according to the gardener guideline gardener-extension-provider-alicloud#507
  • provider-aws -> Adapt the priority according to the gardener guideline gardener-extension-provider-aws#571
  • provider-azure -> Adapt the priority according to the gardener guideline gardener-extension-provider-azure#533
  • provider-equinix-metal -> Adapt the priority according to the gardener guideline gardener-extension-provider-equinix-metal#217
  • provider-gcp -> Adapt the priority according to the gardener guideline gardener-extension-provider-gcp#462
  • provider-openstack -> Adapt the priority according to the gardener guideline gardener-extension-provider-openstack#468
  • provider-vsphere -> Adapt the priority according to the gardener guideline gardener-extension-provider-vsphere#262
  • runtime-gvisor -> Adapt the priority according to the gardener guideline gardener-extension-runtime-gvisor#48
  • shoot-cert-service -> Adapt the priority according to the gardener guideline gardener-extension-shoot-cert-service#126
  • shoot-dns-service -> Adapt the priority according to the gardener guideline gardener-extension-shoot-dns-service#143
  • shoot-networking-filter -> Adapt the priority according to the gardener guideline gardener-extension-shoot-networking-filter#22
  • shoot-networking-problemdetector -> Adapt the priority according to the gardener guideline gardener-extension-shoot-networking-problemdetector#12
  • gardener-extension-shoot-oidc-service -> Adapt the deployment priorityClassName according to the gardener documentation gardener-extension-shoot-oidc-service#69
• terraformer -> Use PriorityClass for terraformer #6515
• drop deprecated gardener-shoot-controlplane once all components have been migrated to the other fine-granular PriorityClasses -> Cleanup old priority class and related migration code  #6738, Drop the gardener-shoot-controlplane PriorityClass #6899

Why is this needed:
These are components with high importance for a gardener landscape and shouldn't be evicted by pods with lower importance that have higher pod priority.

[ialidzhikov]

/assign

[ialidzhikov]

@vpnachev, really nice finding! Thank you for this issue!

[acumino]

Maybe we can use this new feature. [Add non-preempting option to PriorityClasses] (kubernetes/enhancements#902.)

[rfranzke]

/unassign @ialidzhikov
/assign @rfranzke

I still plan to take a look and at least come up with a proposal which PriorityClasses should be introduced and how the components should be assigned (either tomorrow or beginning of next week).

[rfranzke]

We discussed out of band that our usage of PriorityClasses is quite confusing today and that we should think whether we can introduce some better concepts to improve the overall robustness of the system.

I looked into this topic and I propose that we introduce the following PriorityClasses and with their respective associations:

Seed Clusters

The seed controller in gardenlet would create the following to-be "well-known" PriorityClasses as part of the seed bootstrapping. Extensions can rely on them/their names.

Name	Priority	Associated Components
gardener-system-critical	999998950	gardenlet, gardener-resource-manager, istio-ingressgateway, istiod
gardener-system-900	999998900	Extensions, gardener-seed-admission-controller, reversed-vpn-auth-server
gardener-system-800	999998800	dependency-watchdog-endpoint, dependency-watchdog-probe, etcd-druid, (auditlog-)mutator, vpa-admission-controller
gardener-system-700	999998700	auditlog-seed-controller, hvpa-controller, vpa-recommender, vpa-updater
gardener-system-600	999998600	aggregate-alertmanager, alertmanager, fluent-bit, grafana, kube-state-metrics, nginx-ingress-controller, nginx-k8s-backend, prometheus, seed-prometheus, vpa-exporter
gardener-reserve-excess-capacity	-5	reserve-excess-capacity (ref)
gardener-system-500	999998500	etcd-events, etcd-main, kube-apiserver
gardener-system-400	999998400	gardener-resource-manager
gardener-system-300	999998300	cloud-controller-manager, cluster-autoscaler, csi-driver-controller, kube-controller-manager, kube-scheduler, machine-controller-manager, terraformer, vpn-seed-server
gardener-system-200	999998200	csi-snapshot-controller, csi-snapshot-validation, cert-controller-manager, shoot-dns-service, vpa-admission-controller, vpa-recommender, vpa-updater
gardener-system-100	999998100	alertmanager, grafana-operators, grafana-users, kube-state-metrics, prometheus, loki

Shoot Clusters

The shoot controller in gardenlet would create the following to-be "well-known" PriorityClasses as part of the shoo reconciliation. Extensions can rely on them/their names.

Name	Priority	Associated Components
system-node-critical (created by Kubernetes)	2000001000	calico-node, kube-proxy, apiserver-proxy, csi-driver, egress-filter-applier
system-cluster-critical (created by Kubernetes)	2000000000	calico-typha, calico-kube-controllers, coredns, vpn-shoot
gardener-shoot-system-900	999999900	node-problem-detector
gardener-shoot-system-800	999999800	calico-typha-horizontal-autoscaler, calico-typha-vertical-autoscaler
gardener-shoot-system-700	999999700	blackbox-exporter, node-exporter
gardener-shoot-system-600	999999600	addons-nginx-ingress-controller, addons-nginx-ingress-k8s-backend, kubernetes-dashboard, kubernetes-metrics-scraper

Any opinions on this proposal? Let me know if I forgot/missed a component in the tables above.

[timebertt]

I've updated @rfranzke's proposal after another round of discussion and will start working on it this week.
/unassign @rfranzke
/assign @timebertt

[timebertt]

One more nit, I noticed in the above proposal:
gardener-system-critical (like all other classes in Seeds) should have less priority than all gardener-shoot-system-* classes, to ensure shoot cluster critical components have higher priority than seed cluster components if we are on a ManagedSeed.
Therefore, I propose to use gardener-system-critical=999998950.

Additionally, I propose to drop gardener-shoot-system-critical in favor of reusing system-cluster-critical for coredns and vpn-shoot and system-node-critical for egress-filter-applier.

WDYT?

[rfranzke]

Therefore, I propose to use gardener-system-critical=999998950.

Yes, sounds good, I think we talked about this in our meeting but didn't adapt the values accordingly.

Additionally, I propose to drop gardener-shoot-system-critical in favor of reusing system-cluster-critical for coredns and vpn-shoot and system-node-critical for egress-filter-applier.

Fine for me, although this will create a slight asymmetry 😉

[timebertt]

/unassign
I have completed the first steps in #6186 and listed the remaining steps in the original issue description.

[shafeeqes]

/close
All tasks are completed.

[gardener-prow]

@shafeeqes: Closing this issue.

In response to this:

/close
All tasks are completed.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[timebertt]

/reopen

I just observed that a kube-scheduler pod was preempted by a loki pod, so I noticed that some components were not adapted yet:

$ k get po -o=custom-columns="name:.metadata.name,priorityclass:.spec.priorityClassName"
name                                                   priorityclass
cert-controller-manager-5c647d9bc9-56cwv               gardener-system-200
cloud-controller-manager-86bf56fdcf-bqs6h              gardener-system-300
cluster-autoscaler-5bbbbc9b-vx27w                      <none>
csi-driver-controller-7ffbfdffc-tptkg                  gardener-system-300
csi-snapshot-controller-7595464c96-z8ldr               gardener-system-200
csi-snapshot-validation-859b787d9c-4qf4k               <none>
etcd-events-0                                          gardener-system-500
etcd-main-0                                            gardener-system-500
event-logger-66fd786cdc-jfh6n                          gardener-system-100
gardener-resource-manager-64785d9c95-596t9             gardener-system-400
gardener-resource-manager-64785d9c95-ccms6             gardener-system-400
gardener-resource-manager-64785d9c95-dvbrp             gardener-system-400
grafana-operators-76bb69d65c-r7xnk                     gardener-system-100
grafana-users-8584f55bc5-jsmlh                         gardener-system-100
kube-apiserver-78f7d66fcf-j449b                        gardener-system-500
kube-apiserver-78f7d66fcf-jxgt2                        gardener-system-500
kube-apiserver-78f7d66fcf-wc6rl                        gardener-system-500
kube-controller-manager-6d8697459b-t2czj               gardener-system-300
kube-scheduler-7946565f8f-7r82r                        <none>
kube-state-metrics-74c5544c64-sz8qt                    gardener-system-100
loki-0                                                 gardener-system-100
machine-controller-manager-7bdd655b8f-lqx82            gardener-system-300
network-problem-detector-controller-6f895cfd9b-9vq87   gardener-system-200
prometheus-0                                           gardener-system-100
shoot-dns-service-5cb9bf6c89-mrshz                     <none>
vpa-admission-controller-5bfd4b6d86-nrtt6              gardener-system-200
vpa-recommender-54bffb8dc4-8th7j                       gardener-system-200
vpa-updater-cff9dd4cc-27xvp                            gardener-system-200
vpn-seed-server-699ccc8846-n8ws4                       gardener-system-300

Most importantly, the following components that are configured in this repository were not adapted:

• cluster-autoscaler
• kube-scheduler

[gardener-prow]

@timebertt: Reopened this issue.

In response to this:

/reopen

I just observed that a kube-scheduler pod was preempted by a loki pod, so I noticed that some components were not adapted yet:

$ k get po -o=custom-columns="name:.metadata.name,priorityclass:.spec.priorityClassName"
name                                                   priorityclass
cert-controller-manager-5c647d9bc9-56cwv               gardener-system-200
cloud-controller-manager-86bf56fdcf-bqs6h              gardener-system-300
cluster-autoscaler-5bbbbc9b-vx27w                      <none>
csi-driver-controller-7ffbfdffc-tptkg                  gardener-system-300
csi-snapshot-controller-7595464c96-z8ldr               gardener-system-200
csi-snapshot-validation-859b787d9c-4qf4k               <none>
etcd-events-0                                          gardener-system-500
etcd-main-0                                            gardener-system-500
event-logger-66fd786cdc-jfh6n                          gardener-system-100
gardener-resource-manager-64785d9c95-596t9             gardener-system-400
gardener-resource-manager-64785d9c95-ccms6             gardener-system-400
gardener-resource-manager-64785d9c95-dvbrp             gardener-system-400
grafana-operators-76bb69d65c-r7xnk                     gardener-system-100
grafana-users-8584f55bc5-jsmlh                         gardener-system-100
kube-apiserver-78f7d66fcf-j449b                        gardener-system-500
kube-apiserver-78f7d66fcf-jxgt2                        gardener-system-500
kube-apiserver-78f7d66fcf-wc6rl                        gardener-system-500
kube-controller-manager-6d8697459b-t2czj               gardener-system-300
kube-scheduler-7946565f8f-7r82r                        <none>
kube-state-metrics-74c5544c64-sz8qt                    gardener-system-100
loki-0                                                 gardener-system-100
machine-controller-manager-7bdd655b8f-lqx82            gardener-system-300
network-problem-detector-controller-6f895cfd9b-9vq87   gardener-system-200
prometheus-0                                           gardener-system-100
shoot-dns-service-5cb9bf6c89-mrshz                     <none>
vpa-admission-controller-5bfd4b6d86-nrtt6              gardener-system-200
vpa-recommender-54bffb8dc4-8th7j                       gardener-system-200
vpa-updater-cff9dd4cc-27xvp                            gardener-system-200
vpn-seed-server-699ccc8846-n8ws4                       gardener-system-300

Most importantly, the following components that are configured in this repository were not adapted:

• cluster-autoscaler
• kube-scheduler

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rfranzke]

/unassign @acumino
/assign @ialidzhikov

[ialidzhikov]

• cluster-autoscaler and kube-scheduler -> Set priority class for kube-scheduler and cluster-autoscaler #6838
• csi-snapshot-validation -> Set priority class for csi-snapshot-validation gardener-extension-provider-gcp#507 (I found it missing only for provider-gcp, for the other provider extensions it was set)
• shoot-dns-service -> Set priority class for shoot-dns-service gardener-extension-shoot-dns-service#163

[ialidzhikov]

/close
as the items from #5634 (comment) are completed

[gardener-prow]

@ialidzhikov: Closing this issue.

In response to this:

/close
as the items from #5634 (comment) are completed

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[rfranzke]

/reopen
for cleanup
due to #6799

[gardener-prow]

@rfranzke: Reopened this issue.

In response to this:

/reopen
for cleanup
due to #6799

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[ialidzhikov]

/close
as #6899 is merged

[gardener-prow]

@ialidzhikov: Closing this issue.

In response to this:

/close
as #6899 is merged

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.