[Backport release-1.24] etcd snapshot cleanup fails if node name changes

[rancherbot]

This is a backport issue for #3714, automatically created via rancherbot by @vitorsavian

Original issue description:

Environmental Info:

RKE2 Version:
rke2 version v1.21.14+rke2r1 (514ae51)
go version go1.16.14b7

Node(s) CPU architecture, OS, and Version:

Linux testmachine 5.15.70-flatcar rancher/rancher#1 SMP Thu Oct 27 12:53:14 -00 2022 x86_64 Intel(R) Xeon(R) Silver 4210R CPU @ 2.40GHz GenuineIntel GNU/Linux

Cluster Configuration:
We have multiple rke2 clusters, but all of them have at least 3 control plane nodes and multiple workers

Describe the bug:
We have multiple rke2 clusters and all of them have automatic etcd snapshots enabled (taken every 5 hours). We also configured s3 uploading of those snapshots. Recently, we found that no s3 snapshots are uploaded anymore. We investigated the issue and found the following rke2-server output:

Dec 14 05:27:01 testmachine[1665]: time="2022-12-14T05:27:01Z" level=error msg="failed to save local snapshot data to configmap: ConfigMap \"rke2-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"

I checked the code and found that rke2 is leveraging the etcd snapshot capabilities from k3s for this. A function is executed periodically on all control plane nodes. The function takes local snapshots, uploads them to s3 (if configured) and also reconciles a configmap which contains all snapshots and metadata about them. Looking at the code it seems that the reconcilation of that "sync" configmap is based on the name of the node which executes the etcd snapshot. Same goes for the s3 retention functions (only old objects which contain the node name will be cleaned up). As we are replacing all our nodes in the clusters whenever there is a new flatcar version, the node names change quite often. This leads to orphaned entries in the config map and also orphaned objects in the s3 buckets (although this could be worked around with a lifecycle policy).

Are there any ideas what could be done to fix this?

I found this bug report which describes the too large configmap in the rancher repo.

Steps To Reproduce:

Enable etcd snapshots and s3 uploading. After replacing the control plane nodes with new machines (new names), there will be orphaned entries in the 'rke2-etcd-snapshots' configmap. Whenever the configmap grew too large, no new snapshots will be uploaded to s3 anymore.

Expected behavior:
The sync configmap only contains the snapshots of the current nodes of the clusters and removes all other ones.

[vitorsavian]

We are not bumping the k3s version 1.24 on rke2, so I will close this issue and maintain the 1.25 and 1.26.

[vitorsavian]

The version bump PR is merged, so this issue is ready to test it.

• [Release 1.24] Bumping k3s version 1.24 #4680

[aganesh-suse]

Validated on release-1.24 branch with commit e0f9f80

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

cat /etc/os-release | grep PRETTY
PRETTY_NAME="Ubuntu 22.04.2 LTS"

Cluster Configuration:

Server config: 3 etcd, control planes servers/1 agent config

Config.yaml:

Main ETCD SERVER (+CONTROL PLANE) CONFIG:

token: blah
node-name: "server1"
etcd-snapshot-retention: 2
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: xxx
etcd-s3-secret-key: xxx
etcd-s3-bucket: s3-bucket-name
etcd-s3-folder: rke2snap/commit-setup
etcd-s3-region: us-east-2
write-kubeconfig-mode: "0644"

Sample Secondary Etcd, control plane config.yaml:

token: blah
server: https://x.x.x.x:9345
node-name: "server3"
write-kubeconfig-mode: "0644"

AGENT CONFIG:

token: blah
server: https://x.x.x.x:9345
node-name: "agent1"

Testing Steps

1. Create config dir and place the config.yaml file in server/agent nodes:

$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2

Note: First round node-names:
<version|commit>-server1
server2
server3
agent1
2. Install RKE2:
Validate issue Using Commit:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='e0f9f80cbf19d33aa3ebcc050ae4595584ebe74b' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

Reproduce issue Using Version:

curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_VERSION='v1.24.16+rke2r1' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -

3. Wait for 2 minutes.
   Note: The snapshot gets created every 1 minute (etcd-snapshot-schedule-cron: "* * * * *") . Retention is for 2 snapshots (etcd-snapshot-retention: 2).
   Reference for cron job format: https://cloud.google.com/scheduler/docs/configuring/cron-job-schedules
   After 2 minutes: 2 snapshots are created with name etcd-snapshot-server1-2-xxxx if node-name: server1-2 in config.yaml),
4. Check outputs of:

sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots

sudo rke2 etcd-snapshots list

4a. Also check the s3 bucket/folder in aws to see the snapshots listed.
5. Update the node-name in the config.yaml:
node-names:
<version|commit>-server1-<|suffix1>
server2-<|suffix1>
server3-<|suffix1>
agent1-<|suffix1>
6. restart the rke2 service for all nodes.

sudo systemctl restart rke2-server

7. Wait for 2 more minutes and check the snapshot list:

sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots

sudo rke2 etcd-snapshots list

7a. Also check the s3 bucket/folder in aws to see the snapshots listed.

8. Repeat steps 5 through 7 once more.
   node names:
   <version|commit>-server1-<|suffix2>-<|suffix1>
   server2-<|suffix2>-<|suffix1>
   server3-<|suffix2>-<|suffix1>
   agent1-<|suffix2>-<|suffix1>

Replication Results:

• rke2 version used for replication:

SETUP:

$ rke2 -v
rke2 version v1.24.16+rke2r1 (45f561423ea1809559ae1bff999a63bd14fbe2ca)
go version go1.20.6 X:boringcrypto

Node names in order of update for the main etcd server:

version-setup-server1
version-setup-server1-27470           
version-setup-server1-1234-27470

Final output - after multiple node name changes:

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots 
total 67424
-rw------- 1 root root  8478752 Aug 24 19:34 etcd-snapshot-version-setup-server1-1692905644
-rw------- 1 root root  8675360 Aug 24 19:35 etcd-snapshot-version-setup-server1-1692905702
-rw------- 1 root root 10977312 Aug 24 19:37 etcd-snapshot-version-setup-server1-27470-1692905821
-rw------- 1 root root 12394528 Aug 24 19:38 etcd-snapshot-version-setup-server1-27470-1692905883
-rw------- 1 root root 13631520 Aug 24 19:39 etcd-snapshot-version-setup-server1-1234-27470-1692905943
-rw------- 1 root root 14860320 Aug 24 19:40 etcd-snapshot-version-setup-server1-1234-27470-1692906004

$ sudo rke2 etcd-snapshot list 
time="2023-08-24T19:40:58Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-08-24T19:40:58Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-08-24T19:40:58Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-08-24T19:40:58Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-08-24T19:40:58Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-08-24T19:40:58Z" level=info msg="S3 bucket xxxx exists"
Name                                                      Size     Created
etcd-snapshot-version-setup-server1-1234-27470-1692905943 13631520 2023-08-24T19:39:28Z
etcd-snapshot-version-setup-server1-1234-27470-1692906004 14860320 2023-08-24T19:40:06Z
etcd-snapshot-version-setup-server1-1692905644            8478752  2023-08-24T19:34:05Z
etcd-snapshot-version-setup-server1-1692905702            8675360  2023-08-24T19:35:03Z
etcd-snapshot-version-setup-server1-27470-1692905821      10977312 2023-08-24T19:37:02Z
etcd-snapshot-version-setup-server1-27470-1692905883      12394528 2023-08-24T19:38:05Z

As we can see above, previous snapshots with different node-names are still listed and not cleaned up.

Validation Results:

• rke2 version used for validation:

rke2 -v
rke2 version v1.24.16-dev+e0f9f80c (e0f9f80cbf19d33aa3ebcc050ae4595584ebe74b)
go version go1.20.6 X:boringcrypto

Node names in order of update for the main etcd server:

commit-setup-server1
commit-setup-server1-816            
commit-setup-server1-6518-816     

After updating node-names 2 times, the snapshots listed are:

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots 
total 21060
-rw------- 1 root root 10223648 Aug 24 20:34 etcd-snapshot-commit-setup-server1-6518-816-1692909243
-rw------- 1 root root 11333664 Aug 24 20:35 etcd-snapshot-commit-setup-server1-6518-816-1692909302

$ sudo rke2 etcd-snapshot list 
time="2023-08-24T20:35:50Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-08-24T20:35:50Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-08-24T20:35:50Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-08-24T20:35:50Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-08-24T20:35:50Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-08-24T20:35:50Z" level=info msg="S3 bucket xxxx exists"
Name                                                   Size     Created
etcd-snapshot-commit-setup-server1-6518-816-1692909243 10223648 2023-08-24T20:34:05Z
etcd-snapshot-commit-setup-server1-6518-816-1692909302 11333664 2023-08-24T20:35:04Z

As we can see, the previous snapshots with old node-names are no longer retained and get cleaned up.