kubelet: add setting for configuring systemReserved

[gthao313]

Issue number:
Part of #1447

Description of changes:
Adds a new settings kubernetes.system-reserved for configuring

Testing done:
launching instance with the AMI which contains new setting and check if the resources (cpu, memory, storage) have been reserved.
Step1: Check if configuration is here.

cat etc/kubernetes/kubelet/config
systemReserved:
  cpu: "40m"
  ephemeral-storage: "1Gi"
  memory: "1000Mi"

[tianhg@ip-172-31-39-243 ~]$ NODE_NAME="ip-192-168-28-205.us-west-2.compute.internal"
[tianhg@ip-172-31-39-243 ~]$ curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}

ls  kubelet_configz_${NODE_NAME}

  "systemReserved": {
    "cpu": "40m",
    "ephemeral-storage": "1Gi",
    "memory": "1000Mi"
  },

Step2: go to EKS check if the resources (cpu, memory, storage) have been reserved
NodeAllocatable = NodeCapacity - Kube-reserved - system-reserved - eviction-threshold

cpu

[settings.kubernetes.system-reserved.cpu]
cpu = "40m,"
Allocatable cpu = 4000m (NodeCapacity) - 80m (kube-reserved) -40m ( system-reserved)

EKS

Name	Capacity	Allocatable
CPU	4	3880m

memory

[settings.kubernetes.system-reserved.memory]
memory = "1000Mi"
Allocatable memory = 16089632Ki (NodeCapacity) - 893Mi (kube-reserved) -1000Mi ( system-reserved)

EKS

Name	Capacity	Allocatable
Memory	16089632Ki	14048800Ki

ephemeral-storage

[settings.kubernetes.system-reserved.ephemeral-storage]
ephemeral-storage = "1Gi"
Allocatable ephemeral-storage = 20624592Ki (NodeCapacity) - 1Gi (kube-reserved) -1Gi ( system-reserved)

Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         3880m
  ephemeral-storage:           16860140308

Test cases

the case where the new settings are unset

Step1: Check if configuration is here.

userdata - no system-reserved setting
....
cluster-name = "bottlerocket"
[settings.host-containers.admin]
enabled = true
....

result

cat etc/kubernetes/kubelet.config
...
kubeReserved:
  cpu: "80m"
  memory: "893Mi"
  ephemeral-storage: "1Gi"
resolvConf: "/etc/resolv.conf"
...

[tianhg@ip-172-31-39-243 ~]$ NODE_NAME="ip-192-168-17-160.us-west-2.compute.internal"
[tianhg@ip-172-31-39-243 ~]$ curl -sSL "http://localhost:8001/api/v1/nodes/${NODE_NAME}/proxy/configz" | jq '.kubeletconfig|.kind="KubeletConfiguration"|.apiVersion="kubelet.config.k8s.io/v1beta1"' > kubelet_configz_${NODE_NAME}

cat  kubelet_configz_${NODE_NAME}

....
  "kubeReserved": {
    "cpu": "80m",
    "ephemeral-storage": "1Gi",
    "memory": "893Mi"
  },
  "enforceNodeAllocatable": [
    "pods"
  ],
....

systemReserved has not been configured in kubelet.

Step2: go to EKS check if the resources (cpu, memory, storage) have been reserved

Capacity:
  attachable-volumes-aws-ebs:  25
  cpu:                         4
  ephemeral-storage:           20624592Ki
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      15917600Ki
  pods:                        58
Allocatable:
  attachable-volumes-aws-ebs:  25
  cpu:                         3920m
  ephemeral-storage:           17933882132
  hugepages-1Gi:               0
  hugepages-2Mi:               0
  memory:                      14900768Ki
  pods:                        58

  "evictionHard": {
    "imagefs.available": "15%",
    "memory.available": "100Mi",
    "nodefs.available": "10%",
    "nodefs.inodesFree": "5%"
  },

cpu
Allocatable cpu(3920m) = 4000m (NodeCapacity) - 80m (kube-reserved) - 0 ( system-reserved)

memory
Allocatable memory(14900768Ki) = 15917600Ki (NodeCapacity) - 893Mi/914432Ki (kube-reserved) -0 ( system-reserved) - 100Mi/102400Ki (eviction-hard)

ephemeral-storage
Allocatable ephemeral-storage(17933882132) = 20624592Ki/21119582208 (NodeCapacity) -1Gi/1,073,741,824 (kube-reserved) -0 ( system-reserved) - 10%/2111958220.8 (eviction-hard)

Migration test:

upgrade

Step1: Upgrade to v1.1.2

bash-5.0# updog check-update -a --json
[
  {
    "variant": "aws-k8s-1.19",
    "arch": "x86_64",
    "version": "1.1.2",
    "max_version": "1.1.2",
.....
bash-5.0# updog update -i 1.1.2 -r -n
Starting update to 1.1.2

Step2: Specify new setting system-reservedthrough control container

apiclient set -j '{"kubernetes": {"system-reserved": {"memory":"1000Mi", "cpu":"40m", "ephemeral-storage":"1Gi"}}}'

cat ../<tastore/current/live/settings/kubernetes/system-reserved/
cpu  ephemeral-storage	memory

downgrade

Step1: Check migration binary

ls -al /var/lib/bottlerocket-migrations
-rw-r--r--.  1 root root 500380 Jun  7 23:20 2b743f2f624ed6f54d459d650482758b953d895c9988e13e0d2372905a2df181.migrate_v1.1.2_kubelet-system-reserved.lz4
-rw-r--r--.  1 root root   3283 Jun  7 23:20 38ca59d14c8b36a1c6734e1f659e2c1f27278c960d29697c9660865cbd1a6bf6.manifest.json
-rw-r--r--.  1 root root 500571 Jun  7 23:20 bf86b87721ad0c57441262a3251ef00ac3b950dfddaf1c782c0c6720213bec24.migrate_v1.1.2_kubelet-container-log.lz4

Step2: Downgrade to previous verison

signpost rollback-to-inactive
reboot

Step3: Check if system-reserved have been removed

bash-5.0# ls /var/lib/bottlerocket/datastore/current/live/settings/kubernetes
api-server			  max-pods.setting-generator
authentication-mode		  node-ip
cluster-certificate		  node-ip.setting-generator
cluster-dns-ip			  pod-infra-container-image
cluster-dns-ip.setting-generator  pod-infra-container-image.affected-services
cluster-domain			  pod-infra-container-image.setting-generator
cluster-name			  standalone-mode
max-pods			  static-pods.affected-services

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[gthao313]

Push above update README

[zmrow]

🍄