k3s-etcd-snapshots configmap grows too large when snapshot retention is high

[brandond]

• Tracking issue for [BUG] rke2-etcd-snapshots configmap grows too large when snapshot retention is high rancher/rke2#4495 - affects both projects

Describe the bug
If the number of etcd nodes and snapshot retention count is too high, the k3s-etcd-snapshots configmap will grow too large and eventually the k3s process will be unable to save the configmap as it has grown over 1MB.

[aganesh-suse]

Validated on master branch with commit 6aef26e

Environment Details

Infrastructure

• Cloud
• Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

1 server/ 1 agent

Config.yaml:

 $ cat /etc/rancher/k3s/config.yaml 
token: secret
etcd-snapshot-retention: 5
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: xxxx
etcd-s3-secret-key: xxxx
etcd-s3-bucket: xxxx
etcd-s3-folder: xxxx/k3s1
etcd-s3-region: xxxx

cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

Additional Files: metadata.yaml:
https://gist.githubusercontent.com/aganesh-suse/52c3d6c3d7fe70141fa3a49431ac0032/raw/20039a159ab0f5fce1930f5ec12f6afc2b034784/k3s-etcd-snapshot-extra-metadata.yaml

Testing Steps

1. Copy config.yaml

$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s

2. Install k3s

curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='d2b3219c1a3300fd6fc3f6d2ecd4b08e4cfebf5a' sh -s - server

3. Verify Cluster Status:

kubectl get nodes -o wide
kubectl get pods -A

4. Apply k3s snapshot metadata.yaml file. Save the etcd-snapshot file 5 times.

$ kubectl apply -f metadata.yaml
or 
$ kubectl apply -f https://gist.githubusercontent.com/aganesh-suse/52c3d6c3d7fe70141fa3a49431ac0032/raw/20039a159ab0f5fce1930f5ec12f6afc2b034784/k3s-etcd-snapshot-extra-metadata.yaml
$ for (( I=0; I < 5; I++ )); do sudo k3s etcd-snapshot save ; done

Note that with the above extra metadata, the 3rd or 4th snapshot save will fail in repro env and we dont hit the same in the validation environment.
5. Verify etcd snapshot operations:

sudo k3s etcd-snapshot save
sudo k3s etcd-snapshot list
sudo k3s etcd-snapshot prune --snapshot-retention <retention_count>
sudo k3s etcd-snapshot delete <snapshot_name>

Replication Results:

• k3s version used for replication:

 $ k3s -v 
k3s version v1.28.2+k3s1 (6330a5b4)
go version go1.20.8

$ sudo k3s etcd-snapshot save 
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T02:59:15Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
time="2023-10-23T02:59:15Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2023-10-23T02:59:15Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
time="2023-10-23T02:59:15Z" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2023-10-23T02:59:15Z" level=info msg="Creating k3s-supervisor event broadcaster"
time="2023-10-23T02:59:15Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029955"
{"level":"info","ts":"2023-10-23T02:59:15.839694Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029955.part"}
{"level":"info","ts":"2023-10-23T02:59:15.844031Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-23T02:59:15.844136Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-23T02:59:16.078536Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-23T02:59:16.116518Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"18 MB","took":"now"}
{"level":"info","ts":"2023-10-23T02:59:16.116611Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029955"}
time="2023-10-23T02:59:16Z" level=fatal msg="failed to save local snapshot data to configmap: ConfigMap \"k3s-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"

Validation Results:

• k3s version used for validation:

 $ k3s -v 
k3s version v1.28.3-rc1+k3s1 (6aef26e9)
go version go1.20.10

ETCD Snapshot Operations:
Save:

 $ sudo k3s etcd-snapshot save 
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T02:59:27Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029967"
{"level":"info","ts":"2023-10-23T02:59:27.287209Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029967.part"}
{"level":"info","ts":"2023-10-23T02:59:27.289265Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-23T02:59:27.289305Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-23T02:59:27.580246Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-23T02:59:27.620534Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"18 MB","took":"now"}
{"level":"info","ts":"2023-10-23T02:59:27.624366Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029967"}
time="2023-10-23T02:59:27Z" level=info msg="Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-ip-x-x-x-x-1698029967"
time="2023-10-23T02:59:27Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-23T02:59:27Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-23T02:59:27Z" level=info msg="Saving etcd snapshot on-demand-ip-x-x-x-x-1698029967 to S3"
time="2023-10-23T02:59:27Z" level=info msg="Uploading snapshot to s3://xxxx//var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029967"
time="2023-10-23T02:59:28Z" level=info msg="Uploaded snapshot metadata s3://xxxx//var/lib/rancher/k3s/server/db/.metadata/on-demand-ip-x-x-x-x-1698029967"
time="2023-10-23T02:59:28Z" level=info msg="S3 upload complete for on-demand-ip-x-x-x-x-1698029967"
time="2023-10-23T02:59:28Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-23T02:59:28Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

List:

$ sudo k3s etcd-snapshot list 2>/dev/null 
Name                                     Location                                                                                       Size     Created
etcd-snapshot-ip-x-x-x-x-1698029642 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029642       4059168  2023-10-23T02:54:02Z
etcd-snapshot-ip-x-x-x-x-1698029642 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029642 4059168  2023-10-23T02:54:02Z
etcd-snapshot-ip-x-x-x-x-1698029704 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029704       4157472  2023-10-23T02:55:04Z
etcd-snapshot-ip-x-x-x-x-1698029704 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029704 4157472  2023-10-23T02:55:04Z
etcd-snapshot-ip-x-x-x-x-1698029822 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029822       4403232  2023-10-23T02:57:02Z
etcd-snapshot-ip-x-x-x-x-1698029822 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029822 4403232  2023-10-23T02:57:02Z
etcd-snapshot-ip-x-x-x-x-1698029881 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029881       6107168  2023-10-23T02:58:01Z
etcd-snapshot-ip-x-x-x-x-1698029881 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029881 6107168  2023-10-23T02:58:01Z
on-demand-ip-x-x-x-x-1698029928     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029928           7077920  2023-10-23T02:58:48Z
on-demand-ip-x-x-x-x-1698029928     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029928     7077920  2023-10-23T02:58:48Z
on-demand-ip-x-x-x-x-1698029938     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029938           8523808  2023-10-23T02:58:58Z
on-demand-ip-x-x-x-x-1698029938     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029938     8523808  2023-10-23T02:58:58Z
etcd-snapshot-ip-x-x-x-x-1698029945 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029945       10629152 2023-10-23T02:59:05Z
etcd-snapshot-ip-x-x-x-x-1698029945 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029945 10629152 2023-10-23T02:59:05Z
on-demand-ip-x-x-x-x-1698029948     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029948           13791264 2023-10-23T02:59:08Z
on-demand-ip-x-x-x-x-1698029948     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029948     13791264 2023-10-23T02:59:08Z
on-demand-ip-x-x-x-x-1698029958     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029958           16564256 2023-10-23T02:59:18Z
on-demand-ip-x-x-x-x-1698029958     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029958     16564256 2023-10-23T02:59:18Z
on-demand-ip-x-x-x-x-1698029967     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029967           18030624 2023-10-23T02:59:27Z
on-demand-ip-x-x-x-x-1698029967     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029967     18030624 2023-10-23T02:59:27Z

Prune:

 $ sudo k3s etcd-snapshot prune --snapshot-retention 3 
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T03:00:07Z" level=info msg="Applying snapshot retention=3 to local snapshots with prefix on-demand in /var/lib/rancher/k3s/server/db/snapshots"
time="2023-10-23T03:00:07Z" level=info msg="Removing local snapshot /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029938"
time="2023-10-23T03:00:07Z" level=info msg="Removing local snapshot /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029928"
time="2023-10-23T03:00:07Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-23T03:00:07Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-23T03:00:07Z" level=info msg="Applying snapshot retention=3 to snapshots stored in s3://xxxx/xxxx/k3s2/on-demand"
time="2023-10-23T03:00:07Z" level=info msg="Removing S3 snapshot: s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029938"
time="2023-10-23T03:00:07Z" level=info msg="Removing S3 snapshot: s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029928"
time="2023-10-23T03:00:07Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-23T03:00:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698029928"
time="2023-10-23T03:00:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698029938"
time="2023-10-23T03:00:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698029928"
time="2023-10-23T03:00:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698029938"
time="2023-10-23T03:00:08Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

Delete:

 $ sudo k3s etcd-snapshot delete on-demand-ip-x-x-x-x-1698029948 
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T03:00:41Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-23T03:00:41Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-23T03:00:41Z" level=info msg="Snapshot on-demand-ip-x-x-x-x-1698029948 deleted locally"
time="2023-10-23T03:00:41Z" level=info msg="Snapshot on-demand-ip-x-x-x-x-1698029948 deleted from S3"
time="2023-10-23T03:00:41Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-23T03:00:42Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698029948"
time="2023-10-23T03:00:42Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698029948"
time="2023-10-23T03:00:42Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

EtcdSnapshotFile:

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get etcdsnapshotfile 
NAME                                                    SNAPSHOTNAME                               NODE              LOCATION                                                                                         SIZE       CREATIONTIME
local-etcd-snapshot-ip-x-x-x-x-1698029642-ab2b13   etcd-snapshot-ip-x-x-x-x-1698029642   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029642         4059168    2023-10-23T02:54:02Z
local-etcd-snapshot-ip-x-x-x-x-1698029704-98f16b   etcd-snapshot-ip-x-x-x-x-1698029704   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029704         4157472    2023-10-23T02:55:04Z
local-etcd-snapshot-ip-x-x-x-x-1698029822-a507a0   etcd-snapshot-ip-x-x-x-x-1698029822   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029822         4403232    2023-10-23T02:57:02Z
local-etcd-snapshot-ip-x-x-x-x-1698029881-6a3d64   etcd-snapshot-ip-x-x-x-x-1698029881   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029881         6107168    2023-10-23T02:58:01Z
local-etcd-snapshot-ip-x-x-x-x-1698029945-0a57b5   etcd-snapshot-ip-x-x-x-x-1698029945   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698029945         10629152   2023-10-23T02:59:05Z
local-on-demand-ip-x-x-x-x-1698029928-75178d       on-demand-ip-x-x-x-x-1698029928       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029928             7077920    2023-10-23T02:58:48Z
local-on-demand-ip-x-x-x-x-1698029938-941352       on-demand-ip-x-x-x-x-1698029938       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029938             8523808    2023-10-23T02:58:58Z
local-on-demand-ip-x-x-x-x-1698029948-e8735d       on-demand-ip-x-x-x-x-1698029948       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029948             13791264   2023-10-23T02:59:08Z
local-on-demand-ip-x-x-x-x-1698029958-9be856       on-demand-ip-x-x-x-x-1698029958       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029958             16564256   2023-10-23T02:59:18Z
local-on-demand-ip-x-x-x-x-1698029967-0b25ea       on-demand-ip-x-x-x-x-1698029967       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698029967             18030624   2023-10-23T02:59:27Z
s3-etcd-snapshot-ip-x-x-x-x-1698029642-1a4f7f      etcd-snapshot-ip-x-x-x-x-1698029642   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029642   4059168    2023-10-23T02:54:02Z
s3-etcd-snapshot-ip-x-x-x-x-1698029704-e74fc9      etcd-snapshot-ip-x-x-x-x-1698029704   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029704   4157472    2023-10-23T02:55:04Z
s3-etcd-snapshot-ip-x-x-x-x-1698029822-257214      etcd-snapshot-ip-x-x-x-x-1698029822   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029822   4403232    2023-10-23T02:57:02Z
s3-etcd-snapshot-ip-x-x-x-x-1698029881-99038a      etcd-snapshot-ip-x-x-x-x-1698029881   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029881   6107168    2023-10-23T02:58:01Z
s3-etcd-snapshot-ip-x-x-x-x-1698029945-d950b7      etcd-snapshot-ip-x-x-x-x-1698029945   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698029945   10629152   2023-10-23T02:59:05Z
s3-on-demand-ip-x-x-x-x-1698029928-eba3e8          on-demand-ip-x-x-x-x-1698029928       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029928       7077920    2023-10-23T02:58:48Z
s3-on-demand-ip-x-x-x-x-1698029938-a25043          on-demand-ip-x-x-x-x-1698029938       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029938       8523808    2023-10-23T02:58:58Z
s3-on-demand-ip-x-x-x-x-1698029948-659e75          on-demand-ip-x-x-x-x-1698029948       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029948       13791264   2023-10-23T02:59:08Z
s3-on-demand-ip-x-x-x-x-1698029958-6d1a78          on-demand-ip-x-x-x-x-1698029958       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029958       16564256   2023-10-23T02:59:18Z
s3-on-demand-ip-x-x-x-x-1698029967-13ce45          on-demand-ip-x-x-x-x-1698029967       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698029967       18030624   2023-10-23T02:59:27Z