kubelet: add setting for configuring eventRecordQPS and eventBurst

[gthao313]

Issue number:
#1495

Description of changes:
Adds two new settings kubernetes.event-qps and kubernetes.event-burst for configuring

Testing done:

event-qps

Creating large amount of events at same time, and check how many relevant events have been logged.
Compare two argument values (smaller value and larger value ), the number of event that has been logged by smaller value will smaller than by larger value.

Step1: event-qps = 0 unlimited (larger value)

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml;
kubectl delete -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml --force;

Result: 63 events have been captured and logged.

Step2: event-qps = 1 lowest value (smaller value)

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml;
kubectl delete -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml --force;

Result: 42 events have been captured and logged.

Test result: The number of events by event-qps = 1 is smaller than by event-qps = 0, which means setting kubernetes.event-qps works.

event-burst

Creating large amount of events at same time, and check how many relevant events have been logged.
Compare two argument values (smaller value and larger value ), the number of event that has been logged by smaller value will smaller than by larger value.

Step1: event-burst = 100 (larger value)
Run command Twice immediately.

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml;
kubectl delete -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml --force;

Result: 119 events have been captured and logged.

Step2: event-burst = 1 lowest value (smaller value)
Run command Twice immediately.

kubectl apply -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml;
kubectl delete -f amazonlinux.yaml -f debian.yaml -f ubuntu.yaml -f nginx.yaml -f python.yaml -f busybox.yaml -f golang.yaml -f httpd.yaml -f node.yaml -f redis.yaml -f mysql.yaml -f postgres.yaml --force;

Result: 64 events have been captured and logged.

Test result: The number of events by event-burst = 1 is smaller than by event-burst = 100, which means setting kubernetes.event-burst works.

Migration test:

upgrade

Step1: Upgrade to v1.1.0

bash-5.0# updog check-update -a --json
[
  {
    "variant": "aws-k8s-1.19",
    "arch": "x86_64",
    "version": "1.1.0",
    "max_version": "1.1.0",
.....
bash-5.0# updog update -i 1.1.0 -r -n
Starting update to 1.1.0

Step2: Specify new setting event-qps and event-burst through control container

apiclient set -j '{"kubernetes": {"event-qps": 50, "event-burst": 100}}

bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/event-qps
50
bash-5.0# cat /var/lib/bottlerocket/datastore/current/live/settings/kubernetes/event-burst
100

downgrade

Step1: Check migration binary

ls -al /var/lib/bottlerocket-migrations
total 1980
drwx------.  2 root root   4096 Apr 29 22:19 .
drwxr-xr-x. 15 root root   4096 Apr 29 22:20 ..
-rw-r--r--.  1 root root 500376 Apr 29 22:19 20bf88b0fc969d079323303636d59f7dee19334d5330ca4d98f03e83dd0768bc.migrate_v1.1.0_kubelet-server-tls-bootstrap.lz4
-rw-r--r--.  1 root root 500611 Apr 29 22:19 2efee01e95b64359eb0a3be08b6be219ba9401b0e5dd724a6c08551cc09fe72f.migrate_v1.1.0_kubelet-event-qps-event-burst.lz4
-rw-r--r--.  1 root root 500441 Apr 29 22:19 4a5d86d8b8342b562e43ad8aae876bf71249510ec6e1a83b52df998e7180299e.migrate_v1.1.0_kubelet-registry-qps-registry-burst.lz4
-rw-r--r--.  1 root root 500460 Apr 29 22:19 81d7535ebd76e9307766c658f1aaffdba91f227d31acb2421058a9d7ba7a53cb.migrate_v1.1.0_kubelet-cloud-provider.lz4
-rw-r--r--.  1 root root   2964 Apr 29 22:19 f77deadeecd4baa00a7f8182477306f55f9ab6458f77adf336b6e172599f5428.manifest.json

Step2: Downgrade to previous verison

signpost rollback-to-inactive
reboot

Step3: Check if event-qps and event-burst have been removed

bash-5.0# ls /var/lib/bottlerocket/datastore/current/live/settings/kubernetes
api-server			  max-pods.setting-generator
authentication-mode		  node-ip
cluster-certificate		  node-ip.setting-generator
cluster-dns-ip			  pod-infra-container-image
cluster-dns-ip.setting-generator  pod-infra-container-image.affected-services
cluster-domain			  pod-infra-container-image.setting-generator
cluster-name			  standalone-mode
max-pods			  static-pods.affected-services

Terms of contribution:

By submitting this pull request, I agree that this contribution is dual-licensed under the terms of both the Apache License, version 2.0, and the MIT license.

[etungsten]

Can you elaborate a bit more on the migration testing done?
Also there's conflicts that needs to be fixed.

[gthao313]

Push above fix conflicts.

[gthao313]

Push above improve registryPullQPS registryBurst eventRecordQPS and eventBurst kube-config template. Handlebars built-in IF helper treats "0" (zero) as false, so it needs use includeZero flag to make sure IF HELPER treats "0" (zero) as true.

Test
userdata

registry-qps = 0
registry-burst = 0
event-qps = 0
event-burst = 0

kube-config result

registryPullQPS: 0
registryBurst: 0
eventRecordQPS: 0
eventBurst: 0

[gthao313]

Push above fix conflicts