k8s settings: kube-reserved, eviction-hard, cpu-manager-policy, allow-unsafe-sysctls

[gthao313]

My fork draft PR: gthao313#1

Description of changes:

• Add new settings eviction-hard, kube-reserved, allowed-unsafe-sysctls to k8s. User can specify this setting in userdata.

• Enable cpu-manager-policy to static

• Add migration for eviction-hard, kube-reserved, allowed-unsafe-sysctls

example:

[settings.kubernetes.eviction-hard]
"memory.available" = "10Gi"
"nodefs.inodesFree" = “15%”
[settings.kubernetes.kube-reserved]
cpu = "70Mi"
memory = "1Gi"
ephemeral-storage = "1Gi"
[settings.kubernetes]
allowed-unsafe-sysctls = ["net.core.somaxconn", ...]

Testing done:

• Build aws-eks-* images and launched instance.
• Migration test: tested the migration via upgrade and downgrade testing
• Unit tests for new model types EvictionHardKey, KubernetesThresholdValue, ReservedResourcesKey, KubernetesQuantityValue, and CpuManagerPolicy (all passes)

Eviction-hard test

Test step1:
Go to control container run apiclient -u /settings to check if expected setting is there.

Test step2:
ssh to admin contianer and look etc/kubernetes/kubelet/config to check if expected config is there.

evictionHard:
  memory.available: 15%

Test step3 - test effected/behavior:
set up eviction threshold to a high value for triggering eviction.

[evictionHard.memory]
"memory.available" = "99.99%" (high percentage to trigger eviction)

a) Go to EKS to check node condition

MemoryPressure	True	kubelet has insufficient memory available

b) In admin contianer, check kubelet log by journalctl -u kubelet

eviction_manager.go:335] eviction manager: attempting to reclaim memory
eviction_manager.go:346] eviction manager: must evict pod(s) to reclaim memory

kube-reservd test

Test step1:
Go to control container run apiclient -u /settings to check if expected setting is there.

Test step2:
NodeAllocatable = NodeCapacity - Kube-reserved - system-reserved - eviction-threshold
cpu
[settings.kubernetes.kube-reserved.cpu]
cpu = 1
Allocatable cpu = 2 - 1 = 1

EKS Resource allocation:
Name Capacity Allocatable
CPU		2		1

memory
[settings.kubernetes.kube-reserved.Memory]
memory = 1Gi

EKS Resource allocation:
Name    Capacity    Allocatable
Memory	7844780Ki	6693804Ki

ephemeral-storage
[settings.kubernetes.kube-reserved.ephemeral-storage]
ephemeral-storage = 1Gi

Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         2
  ephemeral-storage:           20624592Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      7930796Ki
  pods:                        29
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         1
  ephemeral-storage:           19007623956
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      7828396Ki
  pods:                        29˜

cpu-manager-policy test

Test step1:
Go to host find cpu_manager_state

bash-5.0# cat /var/lib/kubelet/cpu_manager_state
{"policyName":"static","defaultCpuSet":"0-1","checksum":3945352861}bash-5.0#

journalctl -u kubelet
cpu_manager.go:173] [cpumanager] starting with static policy

allowed-unsafe-sysctls test

Test step1:
Go to control container run apiclient -u /settings to check if expected setting is there.

Test step2:
Go to kubelet-config to check if desired result is there.

cat eks/kubernetes/kubelet/config

allowed-unsafe-sysctls = ["net.core.somaxconn", ]

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[gthao313]

Oush above add new model types KubernetesThresholdValue and KubernetesQuantityValue, and those passed unit tests.

[gthao313]

push above fix typo

[gthao313]

Push above improve error mod and syntax.

[gthao313]

Push above enable cpu-manager-policy to static, improve input validation for eviction-hard and kube-reserved, and add more unit test for get_resource_to_reserve_in_range get_cpu_millicores_to_reserve.

[gthao313]

push above fix migration version.

[gthao313]

push above improve KubernetesQuantity regex and improve cpu-manager-policy.

[bcressey]

As an overall test, can you make sure that kubelet-config is still the same for 1.15 + 1.16, 1.17 + 1.18, with 1.19 as an outlier?

❯ sha1sum packages/kubernetes-1.1*/kubelet-config
...

❯ diff -u -p packages/kubernetes-1.1{6,7}/kubelet-config
--- packages/kubernetes-1.16/kubelet-config	2021-03-29 19:27:33.165289932 +0000
+++ packages/kubernetes-1.17/kubelet-config	2021-03-29 19:27:33.165289932 +0000
@@ -36,6 +36,7 @@ cgroupRoot: "/"
 runtimeRequestTimeout: 15m
 featureGates:
   RotateKubeletServerCertificate: true
+  CSIMigration: false
 serializeImagePulls: false
 serverTLSBootstrap: true
 configMapAndSecretChangeDetectionStrategy: Cache

❯ diff -u -p packages/kubernetes-1.1{8,9}/kubelet-config
--- packages/kubernetes-1.18/kubelet-config	2021-03-29 19:27:33.166289937 +0000
+++ packages/kubernetes-1.19/kubelet-config	2021-03-29 19:27:33.166289937 +0000
@@ -42,5 +42,6 @@ serverTLSBootstrap: true
 configMapAndSecretChangeDetectionStrategy: Cache
 tlsCipherSuites:
 - TLS_ECDHE_ECDSA_WITH_AES_128_GCM_SHA256
+volumePluginDir: "/var/lib/kubelet/plugins/volume/exec"
 maxPods: {{default 110 settings.kubernetes.max-pods}}
 staticPodPath: "/etc/kubernetes/static-pods/"

[gthao313]

push above improve variable name and improve kube_reserve_cpu kube_reserve_memory in schnauzer.

[gthao313]

Push above improve number conversion process in kube_reserve_memory and improve eviction-hard in README

Additional Test:
Launching two m5.xlarge instances(one Bottlerocket with my changes & one EKS AL2) and two c5.12xlarge instances(one Bottlerocket with your changes & one EKS AL2) to confirm that we calculate the same values across max pods, CPU reserve, and memory reserve.

Bottlerocket m5 xlarge
kubeReserved:
  cpu: "80m"
  memory: "893Mi"
  ephemeral-storage: "1Gi"

EKS AL2 m5 xlargs
"kubeReserved": {
    "cpu": "80m",
    "ephemeral-storage": "1Gi",
    "memory": "893Mi"
  }

Bottlerocket c5 12.xlarge
kubeReserved:
  cpu: "190m"
  memory: "2829Mi"
  ephemeral-storage: "1Gi"

EKS AL2 c5 12.xlarge
"kubeReserved": {
	"cpu": "190m",
	"ephemeral-storage": "1Gi",
	"memory": "2829Mi"
}

[zmrow]

🍄

[diranged]

This is awesome - we've built an AMI with this patch and its working wonderfully. In a followup PR, we'd like to see the evictionSoft settings exposed, as well as the eviction-max-pod-grace-period setting. Overall though, this is great!

[gthao313]

Hi @diranged. We have opened a new issue about supporting eviction soft related settings. You can follow up here #1445. Thanks!