Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Release-1.26] - k3s-etcd-snapshots configmap grows too large when snapshot retention is high #8613

Closed
brandond opened this issue Oct 12, 2023 · 1 comment
Closed

[Release-1.26] - k3s-etcd-snapshots configmap grows too large when snapshot retention is high #8613

brandond opened this issue Oct 12, 2023 · 1 comment
Assignees
@brandond @aganesh-suse
Milestone
v1.26.10+k3s1

Comments

@brandond
Copy link
Contributor

Backport fix for k3s-etcd-snapshots configmap grows too large when snapshot retention is high
@brandond brandond self-assigned this Oct 12, 2023
@brandond brandond mentioned this issue Oct 12, 2023
Merged
@aganesh-suse aganesh-suse added this to the v1.26.10+k3s1 milestone Oct 13, 2023
@aganesh-suse aganesh-suse self-assigned this Oct 16, 2023
@aganesh-suse
Copy link

Validated on release-1.26 branch with commit 10379aa

Environment Details

Infrastructure

  • Cloud
  • Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

1 server/ 1 agent

Config.yaml:

 $ cat /etc/rancher/k3s/config.yaml 
token: secret
etcd-snapshot-retention: 5
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: xxxx
etcd-s3-secret-key: xxxx
etcd-s3-bucket: xxxx
etcd-s3-folder: xxxx/k3s1
etcd-s3-region: xxxx

cluster-init: true
write-kubeconfig-mode: "0644"
node-external-ip: 1.1.1.1
node-label:
- k3s-upgrade=server

Additional Files: metadata.yaml:
https://gist.githubusercontent.com/aganesh-suse/52c3d6c3d7fe70141fa3a49431ac0032/raw/20039a159ab0f5fce1930f5ec12f6afc2b034784/k3s-etcd-snapshot-extra-metadata.yaml

Testing Steps

  1. Copy config.yaml
$ sudo mkdir -p /etc/rancher/k3s && sudo cp config.yaml /etc/rancher/k3s
  1. Install k3s
curl -sfL https://get.k3s.io | sudo INSTALL_K3S_COMMIT='d2b3219c1a3300fd6fc3f6d2ecd4b08e4cfebf5a' sh -s - server
  1. Verify Cluster Status:
kubectl get nodes -o wide
kubectl get pods -A
  1. Apply k3s snapshot metadata.yaml file. Save the etcd-snapshot file 5 times.
$ kubectl apply -f metadata.yaml
or 
$ kubectl apply -f https://gist.githubusercontent.com/aganesh-suse/52c3d6c3d7fe70141fa3a49431ac0032/raw/20039a159ab0f5fce1930f5ec12f6afc2b034784/k3s-etcd-snapshot-extra-metadata.yaml
$ for (( I=0; I < 5; I++ )); do sudo k3s etcd-snapshot save ; done

Note that with the above extra metadata, the 3rd or 4th snapshot save will fail in repro env and we dont hit the same in the validation environment.
5. Verify etcd snapshot operations:

sudo k3s etcd-snapshot save
sudo k3s etcd-snapshot list
sudo k3s etcd-snapshot prune --snapshot-retention <retention_count>
sudo k3s etcd-snapshot delete <snapshot_name>

Replication Results:

  • k3s version used for replication:
 $ k3s -v 
k3s version v1.26.9+k3s1 (4e217286)
go version go1.20.8

$ sudo k3s etcd-snapshot save 
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T03:35:58Z" level=info msg="Managed etcd cluster bootstrap already complete and initialized"
time="2023-10-23T03:35:58Z" level=info msg="Applying CRD helmcharts.helm.cattle.io"
time="2023-10-23T03:35:59Z" level=info msg="Applying CRD helmchartconfigs.helm.cattle.io"
time="2023-10-23T03:35:59Z" level=info msg="Applying CRD addons.k3s.cattle.io"
time="2023-10-23T03:35:59Z" level=info msg="Creating k3s-supervisor event broadcaster"
time="2023-10-23T03:35:59Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032159"
{"level":"info","ts":"2023-10-23T03:35:59.122376Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032159.part"}
{"level":"info","ts":"2023-10-23T03:35:59.13337Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-23T03:35:59.133505Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-23T03:35:59.421067Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-23T03:35:59.454562Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"16 MB","took":"now"}
{"level":"info","ts":"2023-10-23T03:35:59.454729Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032159"}
time="2023-10-23T03:35:59Z" level=fatal msg="failed to save local snapshot data to configmap: ConfigMap \"k3s-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"

Validation Results:

  • k3s version used for validation:
 $ k3s -v 
k3s version v1.26.10-rc1+k3s1 (10379aa3)
go version go1.20.10

ETCD Snapshot Operations:
Save:

$ sudo k3s etcd-snapshot save 
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T03:36:01Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032161"
{"level":"info","ts":"2023-10-23T03:36:01.17167Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032161.part"}
{"level":"info","ts":"2023-10-23T03:36:01.177904Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-23T03:36:01.178075Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-23T03:36:02.54112Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-23T03:36:02.581466Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"18 MB","took":"1 second ago"}
{"level":"info","ts":"2023-10-23T03:36:02.581729Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032161"}
time="2023-10-23T03:36:02Z" level=info msg="Saving snapshot metadata to /var/lib/rancher/k3s/server/db/.metadata/on-demand-ip-x-x-x-x-1698032161"
time="2023-10-23T03:36:02Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-23T03:36:02Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-23T03:36:02Z" level=info msg="Saving etcd snapshot on-demand-ip-x-x-x-x-1698032161 to S3"
time="2023-10-23T03:36:02Z" level=info msg="Uploading snapshot to s3://xxxx//var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032161"
time="2023-10-23T03:36:03Z" level=info msg="Uploaded snapshot metadata s3://xxxx//var/lib/rancher/k3s/server/db/.metadata/on-demand-ip-x-x-x-x-1698032161"
time="2023-10-23T03:36:03Z" level=info msg="S3 upload complete for on-demand-ip-x-x-x-x-1698032161"
time="2023-10-23T03:36:03Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-23T03:36:03Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032161.part"
time="2023-10-23T03:36:03Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

List:

 $ sudo k3s etcd-snapshot list 2>/dev/null 
Name                                     Location                                                                                       Size     Created
etcd-snapshot-ip-x-x-x-x-1698031920 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698031920       1167392  2023-10-23T03:32:00Z
etcd-snapshot-ip-x-x-x-x-1698031920 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698031920 1167392  2023-10-23T03:32:00Z
etcd-snapshot-ip-x-x-x-x-1698031983 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698031983       1404960  2023-10-23T03:33:03Z
etcd-snapshot-ip-x-x-x-x-1698031983 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698031983 1404960  2023-10-23T03:33:03Z
etcd-snapshot-ip-x-x-x-x-1698032041 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698032041       2789408  2023-10-23T03:34:01Z
etcd-snapshot-ip-x-x-x-x-1698032041 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698032041 2789408  2023-10-23T03:34:01Z
etcd-snapshot-ip-x-x-x-x-1698032103 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698032103       4972576  2023-10-23T03:35:03Z
etcd-snapshot-ip-x-x-x-x-1698032103 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698032103 4972576  2023-10-23T03:35:03Z
on-demand-ip-x-x-x-x-1698032122     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032122           6635552  2023-10-23T03:35:22Z
on-demand-ip-x-x-x-x-1698032122     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032122     6635552  2023-10-23T03:35:22Z
on-demand-ip-x-x-x-x-1698032132     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032132           8736800  2023-10-23T03:35:32Z
on-demand-ip-x-x-x-x-1698032132     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032132     8736800  2023-10-23T03:35:32Z
on-demand-ip-x-x-x-x-1698032142     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032142           12275744 2023-10-23T03:35:42Z
on-demand-ip-x-x-x-x-1698032142     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032142     12275744 2023-10-23T03:35:42Z
on-demand-ip-x-x-x-x-1698032152     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032152           14331936 2023-10-23T03:35:52Z
on-demand-ip-x-x-x-x-1698032152     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032152     14331936 2023-10-23T03:35:52Z
etcd-snapshot-ip-x-x-x-x-1698032161 file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698032161       16588832 2023-10-23T03:36:01Z
on-demand-ip-x-x-x-x-1698032161     file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032161           18407456 2023-10-23T03:36:01Z
etcd-snapshot-ip-x-x-x-x-1698032161 s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698032161 16588832 2023-10-23T03:36:01Z
on-demand-ip-x-x-x-x-1698032161     s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032161     18407456 2023-10-23T03:36:01Z

Prune:

 $ sudo k3s etcd-snapshot prune --snapshot-retention 3 
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T03:36:42Z" level=info msg="Applying snapshot retention=3 to local snapshots with prefix on-demand in /var/lib/rancher/k3s/server/db/snapshots"
time="2023-10-23T03:36:42Z" level=info msg="Removing local snapshot /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032132"
time="2023-10-23T03:36:42Z" level=info msg="Removing local snapshot /var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032122"
time="2023-10-23T03:36:42Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-23T03:36:42Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-23T03:36:42Z" level=info msg="Applying snapshot retention=3 to snapshots stored in s3://xxxx/xxxx/k3s2/on-demand"
time="2023-10-23T03:36:42Z" level=info msg="Removing S3 snapshot: s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032132"
time="2023-10-23T03:36:42Z" level=info msg="Removing S3 snapshot: s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032122"
time="2023-10-23T03:36:42Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-23T03:36:42Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032122"
time="2023-10-23T03:36:42Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032132"
time="2023-10-23T03:36:42Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032122"
time="2023-10-23T03:36:42Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032132"
time="2023-10-23T03:36:43Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

Delete:

 $ sudo k3s etcd-snapshot delete on-demand-ip-x-x-x-x-1698032142 
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --cluster-init found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=warning msg="Unknown flag --node-label found in config.yaml, skipping\n"
time="2023-10-23T03:37:16Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-23T03:37:16Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-23T03:37:16Z" level=info msg="Snapshot on-demand-ip-x-x-x-x-1698032142 deleted locally"
time="2023-10-23T03:37:16Z" level=info msg="Snapshot on-demand-ip-x-x-x-x-1698032142 deleted from S3"
time="2023-10-23T03:37:16Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-23T03:37:17Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032142"
time="2023-10-23T03:37:17Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1698032142"
time="2023-10-23T03:37:17Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

EtcdSnapshotFile:

$ sudo /usr/local/bin/kubectl --kubeconfig /etc/rancher/k3s/k3s.yaml get etcdsnapshotfile 
NAME                                                    SNAPSHOTNAME                               NODE              LOCATION                                                                                         SIZE       CREATIONTIME
local-etcd-snapshot-ip-x-x-x-x-1698031920-e65d23   etcd-snapshot-ip-x-x-x-x-1698031920   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698031920         1167392    2023-10-23T03:32:00Z
local-etcd-snapshot-ip-x-x-x-x-1698031983-04dc54   etcd-snapshot-ip-x-x-x-x-1698031983   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698031983         1404960    2023-10-23T03:33:03Z
local-etcd-snapshot-ip-x-x-x-x-1698032041-2eaff5   etcd-snapshot-ip-x-x-x-x-1698032041   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698032041         2789408    2023-10-23T03:34:01Z
local-etcd-snapshot-ip-x-x-x-x-1698032103-9e36d4   etcd-snapshot-ip-x-x-x-x-1698032103   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698032103         4972576    2023-10-23T03:35:03Z
local-etcd-snapshot-ip-x-x-x-x-1698032161-c0bd2c   etcd-snapshot-ip-x-x-x-x-1698032161   ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1698032161         16588832   2023-10-23T03:36:01Z
local-on-demand-ip-x-x-x-x-1698032122-218544       on-demand-ip-x-x-x-x-1698032122       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032122             6635552    2023-10-23T03:35:22Z
local-on-demand-ip-x-x-x-x-1698032132-da6094       on-demand-ip-x-x-x-x-1698032132       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032132             8736800    2023-10-23T03:35:32Z
local-on-demand-ip-x-x-x-x-1698032142-e65622       on-demand-ip-x-x-x-x-1698032142       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032142             12275744   2023-10-23T03:35:42Z
local-on-demand-ip-x-x-x-x-1698032152-864467       on-demand-ip-x-x-x-x-1698032152       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032152             14331936   2023-10-23T03:35:52Z
local-on-demand-ip-x-x-x-x-1698032161-ddb5d9       on-demand-ip-x-x-x-x-1698032161       ip-x-x-x-x   file:///var/lib/rancher/k3s/server/db/snapshots/on-demand-ip-x-x-x-x-1698032161             18407456   2023-10-23T03:36:01Z
s3-etcd-snapshot-ip-x-x-x-x-1698031920-ce0934      etcd-snapshot-ip-x-x-x-x-1698031920   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698031920   1167392    2023-10-23T03:32:00Z
s3-etcd-snapshot-ip-x-x-x-x-1698031983-60fb7a      etcd-snapshot-ip-x-x-x-x-1698031983   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698031983   1404960    2023-10-23T03:33:03Z
s3-etcd-snapshot-ip-x-x-x-x-1698032041-314f6b      etcd-snapshot-ip-x-x-x-x-1698032041   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698032041   2789408    2023-10-23T03:34:01Z
s3-etcd-snapshot-ip-x-x-x-x-1698032103-ca65ee      etcd-snapshot-ip-x-x-x-x-1698032103   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698032103   4972576    2023-10-23T03:35:03Z
s3-etcd-snapshot-ip-x-x-x-x-1698032161-ffbcba      etcd-snapshot-ip-x-x-x-x-1698032161   ip-x-x-x-x   s3://xxxx/xxxx/k3s2/etcd-snapshot-ip-x-x-x-x-1698032161   16588832   2023-10-23T03:36:01Z
s3-on-demand-ip-x-x-x-x-1698032122-fabefd          on-demand-ip-x-x-x-x-1698032122       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032122       6635552    2023-10-23T03:35:22Z
s3-on-demand-ip-x-x-x-x-1698032132-0a8937          on-demand-ip-x-x-x-x-1698032132       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032132       8736800    2023-10-23T03:35:32Z
s3-on-demand-ip-x-x-x-x-1698032142-540fd3          on-demand-ip-x-x-x-x-1698032142       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032142       12275744   2023-10-23T03:35:42Z
s3-on-demand-ip-x-x-x-x-1698032152-4a1d6a          on-demand-ip-x-x-x-x-1698032152       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032152       14331936   2023-10-23T03:35:52Z
s3-on-demand-ip-x-x-x-x-1698032161-23ab1f          on-demand-ip-x-x-x-x-1698032161       ip-x-x-x-x   s3://xxxx/xxxx/k3s2/on-demand-ip-x-x-x-x-1698032161       18407456   2023-10-23T03:36:01Z

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@aganesh-suse aganesh-suse

Labels
None yet
Projects
K3s Development
Archived in project
Milestone
v1.26.10+k3s1
Development

No branches or pull requests
2 participants
@brandond @aganesh-suse