Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[BUG] rke2-etcd-snapshots configmap grows too large when snapshot retention is high #4495

Closed
ratschance opened this issue Jul 22, 2022 · 9 comments
Closed

[BUG] rke2-etcd-snapshots configmap grows too large when snapshot retention is high #4495

ratschance opened this issue Jul 22, 2022 · 9 comments
Assignees
@brandond @aganesh-suse
Labels
kind/bug Something isn't working
Milestone
v1.28.3+rke2r1

Comments

@ratschance
Copy link

ratschance commented Jul 22, 2022 •
edited by brandond
Loading

Rancher Server Setup

  • Rancher version: 2.6.5
  • Installation option (Docker install/Helm Chart): Helm Chart
    • If Helm Chart, Kubernetes Cluster and version (RKE1, RKE2, k3s, EKS, etc): RKE2
  • Proxy/Cert Details:

Information about the Cluster

  • Kubernetes version: v1.22.9+rke2r2
  • Cluster Type (Local/Downstream): Downstream also on v1.22.9+rke2r2
    • If downstream, what type of cluster?: Custom

User Information

  • What is the role of the user logged in? Admin

Describe the bug
If the number of etcd nodes and snapshot retention count is too high, the rke2-etcd-snapshots configmap will grow too large and eventually the rke2-server process will be unable to save the configmap as it has grown over 1MB.

To Reproduce

  • Create a rke2 cluster with 5 wrangler nodes
  • Enable s3 backups with retention set at 72
  • May need enough cluster configuration to make the metadata field grow.

With 5 wrangler nodes and 72 snapshot retention there are 5 nodes * (1 local + 1 s3) * 72 retained = 720 entries per rke2-etcd-snapshots configmap. Of note, I have noticed that the metadata field on each entry is about 1.1KB, so that's the likely problem here.

Result
At some point snapshots will start failing to update in the configmap. Viewing the logs in rke2-server.service should show

level=error msg="failed to save local snapshot data to configmap: ConfigMap \"rke2-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"

Expected Result
Expected that 72 snapshot retention would work in this case.

Screenshots

Additional context
@ratschance ratschance added the kind/bug Something isn't working label Jul 22, 2022
@thirdeyenick thirdeyenick mentioned this issue Dec 15, 2022
Closed
@mikejoh
Copy link

mikejoh commented Feb 8, 2023 •
edited
Loading

Can confirm that we have the same issue, we're on Rancher v2.6.9.

In our case the retention (set to 15) didn't work since we lacked the correct permissions to delete objects in S3, so we basically had local retention that worked but an ever growing amount of objects in S3. In the midst of this we had two clusters failing to upload snapshot backups to S3 due to the error mentioned in this issue. When we resolved the retention and added the correct permission this error went away.

Note that we had a much lower retention configured than mentioned here.

@mateuszdrab
Copy link

mateuszdrab commented Feb 10, 2023 •
edited
Loading

Same issue.

I bumped my retention from the default to 28 with snapshots taken every 6 hours and a couple of days after snapshots stopped uploading to S3. I actually noticed this only when I needed to restore etcd due to an emergency and luckily, it was still there on the node but not in S3.

Rancher v2.7.0
RKE v1.24.9+rke2r2

Based on etcd-io/etcd#7968, the upper limit in etcd is 1.5Mi, not sure where the 1Mi limitation is coming from - I bumped the limit (https://etcd.io/docs/v3.5/dev-guide/limit/#request-size-limit) to 2Mi and I still get the same error.

Reducing the retention value also makes no difference at the moment.

@guaychou
Copy link

any update on this ?

@snasovich
Copy link

@cwayne18 @caroline-suse-rancher , transferring to RKE2 team as rke2-etcd-snapshots config map is populated by RKE2 itself and it's growing over the limit.

@snasovich snasovich transferred this issue from rancher/rancher Jul 19, 2023
@cwayne18
Copy link
Contributor

@caroline-suse-rancher think this makes sense for @vitorsavian to take next?

@brandond brandond mentioned this issue Jul 28, 2023
Merged
This was referenced Aug 2, 2023
Closed
Closed
Closed
@caroline-suse-rancher caroline-suse-rancher added this to the v1.28.3+rke2r1 milestone Oct 10, 2023
@aganesh-suse aganesh-suse self-assigned this Oct 12, 2023
This was referenced Oct 12, 2023
Closed
Closed
Closed
Closed
Merged
@aganesh-suse
Copy link

aganesh-suse commented Oct 20, 2023 •
edited
Loading

Ubuntu 22.04
HA: 3 server/1 agent

Am unable to reproduce the original issue on a standalone rke2.
I have set the retention from 150 upto 500, and had 3 etcd servers.
Was able to take upto 300+ on-demand snapshots in a loop.
After which there was no space in the VM to store the snapshots, and so the rke2 service had space related errors.
But I did not see any save to configmap errors as reported above.

Some reproducing attempt sample results with:

$ rke2 -v
rke2 version v1.28.2+rke2r1 (7466261e4792e68baa2cc0c2afd3dcc929d72061)
go version go1.20.8 X:boringcrypto

$ sudo rke2 etcd-snapshot save
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --tls-san found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --kube-apiserver-arg found in config.yaml, skipping
WARN[0000] Unknown flag --cni found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Managed etcd cluster bootstrap already complete and initialized
INFO[0000] Applying CRD helmcharts.helm.cattle.io
INFO[0000] Applying CRD helmchartconfigs.helm.cattle.io
INFO[0000] Applying CRD addons.k3s.cattle.io
INFO[0000] Creating rke2-supervisor event broadcaster
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697742786
{"level":"info","ts":"2023-10-19T19:13:06.156092Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697742786.part"}
{"level":"info","ts":"2023-10-19T19:13:06.160944Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-19T19:13:06.160997Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-19T19:13:09.998694Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-19T19:13:10.147455Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"128 MB","took":"3 seconds ago"}
{"level":"info","ts":"2023-10-19T19:13:10.14755Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697742786"}
INFO[0004] Saving etcd snapshot on-demand-ip-x-x-x-x-1697742786 to S3
INFO[0004] Checking if S3 bucket xxxx exists
INFO[0004] S3 bucket xxxx exists
INFO[0004] Uploading snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697742786 to S3
INFO[0006] S3 upload complete for on-demand-ip-x-x-x-x-1697742786
INFO[0006] Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap
INFO[0006] Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete

$ sudo rke2 etcd-snapshot list | wc -l
181

rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-10-19 18:20:23 UTC; 55min ago
       Docs: https://github.com/rancher/rke2#readme
    Process: 681794 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 681796 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 681797 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 681798 (rke2)
      Tasks: 184
     Memory: 1.3G
        CPU: 7min 50.470s
     CGroup: /system.slice/rke2-server.service
             ├─681798 "/usr/local/bin/rke2 server"
             ├─681807 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd
             ├─681848 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/rke>
             ├─681886 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 313cd48c6c055d75fd84cf3e6b1da5efc7c3c4b6d8d9b7fc7ec9daffb2137178 -address /run/k3s/containerd/containerd.sock
             ├─681968 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 4d3e12ae563259f2a8559f16e8ebc1588fa27dc473fd628ba4f6575c3123aebc -address /run/k3s/containerd/containerd.sock
             ├─682060 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id a282117609da51d22ceab43261a6a444539f9672300949f535634cf8005653ec -address /run/k3s/containerd/containerd.sock
             ├─682070 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3a36c7065169a824aef7e3b2eb6470f0d8ef1ddda5f720292d18f35ab238703c -address /run/k3s/containerd/containerd.sock
             ├─682217 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id ebfe8925731b441739283e623f35c57160a8cc2f345a8f5dda4a5bf0ca51d682 -address /run/k3s/containerd/containerd.sock
             ├─682365 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 674c952346e9c85d797b02a6649a888e7e37626e390d5b9e3adb7658b047e94a -address /run/k3s/containerd/containerd.sock
             ├─683007 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6e20b0d885f89ee7dab5d9172e5b7832ce28a4472c5a5a96fe2a9d36c6f628b7 -address /run/k3s/containerd/containerd.sock
             ├─683102 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 446bc52f8d5915e8b4accc637aa2b02e6c3add386372891b7a1443ebb5146f9f -address /run/k3s/containerd/containerd.sock
             ├─685270 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id bc86be4f66c826cb8cabfd7fe8563c8c3783329529c175ca48b5b84e53228e4b -address /run/k3s/containerd/containerd.sock
             ├─686072 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id d912500a28b583c020623603f70e2330bdd9a9dcb618f448503dbe9a68ea633c -address /run/k3s/containerd/containerd.sock
             ├─686861 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id f9ee8c9cc7300790dbc706833b719178fb1a30d9789e6982d81fde61fc6792a8 -address /run/k3s/containerd/containerd.sock
             └─687059 /var/lib/rancher/rke2/data/v1.28.2-rke2r1-7a91505b83c3/bin/containerd-shim-runc-v2 -namespace k8s.io -id 3fb7703cbd61e7281418097ca7f79ab293cf77fbf90b3bb5322ccd60082f7686 -address /run/k3s/containerd/containerd.sock

Oct 19 19:15:04 ip-x-x-x-x rke2[681798]: {"level":"info","ts":"2023-10-19T19:15:04.592201Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
Oct 19 19:15:04 ip-x-x-x-x rke2[681798]: {"level":"info","ts":"2023-10-19T19:15:04.870783Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"128 MB","took":"2 seconds ago"}
Oct 19 19:15:04 ip-x-x-x-x rke2[681798]: {"level":"info","ts":"2023-10-19T19:15:04.870914Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697742902"}
Oct 19 19:15:04 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:04Z" level=info msg="Applying local snapshot retention policy: retention: 150, snapshotPrefix: etcd-snapshot, directory: /var/lib/rancher/rke2/server/db/snapshots"
Oct 19 19:15:04 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:04Z" level=info msg="Saving etcd snapshot etcd-snapshot-ip-x-x-x-x-1697742902 to S3"
Oct 19 19:15:04 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:04Z" level=info msg="Uploading snapshot /var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697742902 to S3"
Oct 19 19:15:06 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:06Z" level=info msg="S3 upload complete for etcd-snapshot-ip-x-x-x-x-1697742902"
Oct 19 19:15:06 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:06Z" level=info msg="Applying snapshot retention policy to snapshots stored in S3: retention: 150, snapshotPrefix: arch-rke2snap/version-setup/rke2-1/etcd-snapshot"
Oct 19 19:15:06 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:06Z" level=info msg="Reconciling etcd snapshot data in rke2-etcd-snapshots ConfigMap"
Oct 19 19:15:06 ip-x-x-x-x rke2[681798]: time="2023-10-19T19:15:06Z" level=info msg="Reconciliation of snapshot data in rke2-etcd-snapshots ConfigMap complete"

Validated on master branch with commit 0d0d0e4

Environment Details

Infrastructure

  • Cloud
  • Hosted

Node(s) CPU architecture, OS, and Version:

$ cat /etc/os-release
PRETTY_NAME="Ubuntu 22.04.2 LTS"

$ uname -m
x86_64

Cluster Configuration:

HA : 3 server / 1 agent

Config.yaml:

$ sudo cat /etc/rancher/rke2/config.yaml
token: secret
etcd-snapshot-retention: 150
etcd-snapshot-schedule-cron: "* * * * *"
etcd-s3: true
etcd-s3-access-key: xxxx
etcd-s3-secret-key: xxxx
etcd-s3-bucket: xxxx
etcd-s3-folder: xxxx
etcd-s3-region: xxxx
node-external-ip: 1.1.1.1

Testing Steps

  1. Copy config.yaml
$ sudo mkdir -p /etc/rancher/rke2 && sudo cp config.yaml /etc/rancher/rke2
  1. Install RKE2
curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_COMMIT='0d0d0e4879fdf95254461e3a49224f75d7b2dc3d' INSTALL_RKE2_TYPE='server' INSTALL_RKE2_METHOD=tar sh -
  1. Start the RKE2 service
$ sudo systemctl enable --now rke2-server
or 
$ sudo systemctl enable --now rke2-agent
  1. Verify Cluster Status:
kubectl get nodes -o wide
kubectl get pods -A
  1. Verify etcd snapshot operations:
sudo rke2 etcd-snapshot list
sudo rke2 etcd-snapshot save
sudo rke2 etcd-snapshot prune --snapshot-retention <value>
sudo rke2 etcd-snapshot delete <snapshot_name>

Validation Results:

  • rke2 version used for validation:
$ rke2 -v
rke2 version v1.28.3+dev.0d0d0e48 (0d0d0e4879fdf95254461e3a49224f75d7b2dc3d)
go version go1.20.10 X:boringcrypto

$ sudo ls -lrt /var/lib/rancher/rke2/server/db/snapshots | grep -v total | wc -l
159
$ sudo rke2 etcd-snapshot list 2>/dev/null | grep -v Name |  wc -l
318

list of etcdsnapshotfile from all 3 servers:

$ /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get etcdsnapshotfile | grep -v NAME | wc -l 
454

Please note results were gathered during active automation run, where parallel snapshots were being processed across 3 servers:

$ /var/lib/rancher/rke2/bin/kubectl --kubeconfig /etc/rancher/rke2/rke2.yaml get cm -n kube-system rke2-etcd-snapshots
NAME                  DATA   AGE
rke2-etcd-snapshots   308    35m

Above results tabulated across 3 servers:

    SERVER1: ls folder = 159; rke2 snapshot list = 318; 
    SERVER2: ls folder = 44; rke2 snapshot list = 88; 
    SERVER3: ls folder = 37; rke2 snapshot list = 75;         
    SERVER1: get etcdsnapshotfile = 454; configmap data = 308
    SERVER2: get etcdsnapshotfile = 452; configmap data = 307
    SERVER3: get etcdsnapshotfile = 451; configmap data = 308

Sample etcd-snapshot operation results:

$ sudo rke2 etcd-snapshot save 
time="2023-10-19T23:37:22Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-19T23:37:22Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-19T23:37:22Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-19T23:37:22Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-19T23:37:22Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-19T23:37:22Z" level=info msg="Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758643"
{"level":"info","ts":"2023-10-19T23:37:22.766672Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758643.part"}
{"level":"info","ts":"2023-10-19T23:37:22.769077Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-19T23:37:22.769115Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-19T23:37:23.010534Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-19T23:37:23.040039Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"13 MB","took":"now"}
{"level":"info","ts":"2023-10-19T23:37:23.040141Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758643"}
time="2023-10-19T23:37:23Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-19T23:37:23Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-19T23:37:23Z" level=info msg="Saving etcd snapshot on-demand-ip-x-x-x-x-1697758643 to S3"
time="2023-10-19T23:37:23Z" level=info msg="Uploading snapshot to s3://xxxx//var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:37:23Z" level=info msg="Uploaded snapshot metadata s3://xxxx//var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:37:23Z" level=info msg="S3 upload complete for on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:37:23Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-19T23:37:23Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697758563"
time="2023-10-19T23:37:23Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758381"
time="2023-10-19T23:37:23Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758443"
time="2023-10-19T23:37:23Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758505"
time="2023-10-19T23:37:23Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758563"
time="2023-10-19T23:37:23Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758625"
time="2023-10-19T23:37:23Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758443"
time="2023-10-19T23:37:23Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758622"
time="2023-10-19T23:37:24Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758383"
time="2023-10-19T23:37:24Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758504"
time="2023-10-19T23:37:24Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758565"
time="2023-10-19T23:37:25Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

$ sudo rke2 etcd-snapshot prune --snapshot-retention 3 
time="2023-10-19T23:38:07Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-19T23:38:07Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-19T23:38:07Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-19T23:38:07Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-19T23:38:07Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-19T23:38:07Z" level=info msg="Applying snapshot retention=3 to local snapshots with prefix on-demand in /var/lib/rancher/rke2/server/db/snapshots"
time="2023-10-19T23:38:07Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758647"
time="2023-10-19T23:38:07Z" level=info msg="Removing local snapshot /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:38:07Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-19T23:38:08Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-19T23:38:08Z" level=info msg="Applying snapshot retention=3 to snapshots stored in s3://xxxx/xxxx/xxxrke2-1/on-demand"
time="2023-10-19T23:38:08Z" level=info msg="Removing S3 snapshot: s3://xxxx/xxxx/xxxrke2-1/on-demand-ip-x-x-x-x-1697758647"
time="2023-10-19T23:38:08Z" level=info msg="Removing S3 snapshot: s3://xxxx/xxxx/xxxrke2-1/on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:38:08Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758647"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697758443"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697758500"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697758563"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697758624"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697758683"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758443"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758505"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758563"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697758625"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758643"
time="2023-10-19T23:38:08Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758647"
time="2023-10-19T23:38:09Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758622"
time="2023-10-19T23:38:09Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

$ sudo rke2 etcd-snapshot delete on-demand-ip-x-x-x-x-1697758649     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758649            13594656 2023-10-19T23:37:29Z
time="2023-10-19T23:38:44Z" level=warning msg="Unknown flag --token found in config.yaml, skipping\n"
time="2023-10-19T23:38:44Z" level=warning msg="Unknown flag --etcd-snapshot-retention found in config.yaml, skipping\n"
time="2023-10-19T23:38:44Z" level=warning msg="Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping\n"
time="2023-10-19T23:38:44Z" level=warning msg="Unknown flag --write-kubeconfig-mode found in config.yaml, skipping\n"
time="2023-10-19T23:38:44Z" level=warning msg="Unknown flag --node-external-ip found in config.yaml, skipping\n"
time="2023-10-19T23:38:44Z" level=info msg="Checking if S3 bucket xxxx exists"
time="2023-10-19T23:38:44Z" level=info msg="S3 bucket xxxx exists"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot on-demand-ip-x-x-x-x-1697758649 deleted locally"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot on-demand-ip-x-x-x-x-1697758649 deleted from S3"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758649 not found locally"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758649 deleted from S3"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot 13594656 not found locally"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot 13594656 deleted from S3"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot 2023-10-19T23:37:29Z not found locally"
time="2023-10-19T23:38:44Z" level=info msg="Snapshot 2023-10-19T23:37:29Z deleted from S3"
time="2023-10-19T23:38:44Z" level=info msg="Reconciling ETCDSnapshotFile resources"
time="2023-10-19T23:38:44Z" level=info msg="Deleting ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758649"
time="2023-10-19T23:38:44Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758443"
time="2023-10-19T23:38:44Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758565"
time="2023-10-19T23:38:44Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758682"
time="2023-10-19T23:38:44Z" level=info msg="Creating ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758651"
time="2023-10-19T23:38:44Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697758504"
time="2023-10-19T23:38:44Z" level=info msg="Creating ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697758654"
time="2023-10-19T23:38:45Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

$ sudo rke2 etcd-snapshot list 2>/dev/null 
Name                                      Location                                                                                          Size     Created
etcd-snapshot-ip-x-x-x-x-1697758443 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697758443        10604576 2023-10-19T23:34:03Z
etcd-snapshot-ip-x-x-x-x-1697758443 s3://xxxx/xxxx/xxxrke2-1/etcd-snapshot-ip-x-x-x-x-1697758443 10604576 2023-10-19T23:34:03Z
etcd-snapshot-ip-x-x-x-x-1697758504 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697758504        11276320 2023-10-19T23:35:04Z
etcd-snapshot-ip-x-x-x-x-1697758504 s3://xxxx/xxxx/xxxrke2-1/etcd-snapshot-ip-x-x-x-x-1697758504 11276320 2023-10-19T23:35:04Z
etcd-snapshot-ip-x-x-x-x-1697758565 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697758565        11866144 2023-10-19T23:36:05Z
etcd-snapshot-ip-x-x-x-x-1697758565 s3://xxxx/xxxx/xxxrke2-1/etcd-snapshot-ip-x-x-x-x-1697758565 11866144 2023-10-19T23:36:05Z
etcd-snapshot-ip-x-x-x-x-1697758622 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697758622        12902432 2023-10-19T23:37:02Z
etcd-snapshot-ip-x-x-x-x-1697758622 s3://xxxx/xxxx/xxxrke2-1/etcd-snapshot-ip-x-x-x-x-1697758622 12902432 2023-10-19T23:37:02Z
on-demand-ip-x-x-x-x-1697758649     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758649            13594656 2023-10-19T23:37:29Z
on-demand-ip-x-x-x-x-1697758649     s3://xxxx/xxxx/xxxrke2-1/on-demand-ip-x-x-x-x-1697758649     13594656 2023-10-19T23:37:29Z
on-demand-ip-x-x-x-x-1697758651     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758651            13627424 2023-10-19T23:37:31Z
on-demand-ip-x-x-x-x-1697758651     s3://xxxx/xxxx/xxxrke2-1/on-demand-ip-x-x-x-x-1697758651     13627424 2023-10-19T23:37:31Z
on-demand-ip-x-x-x-x-1697758654     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697758654            13680672 2023-10-19T23:37:34Z
on-demand-ip-x-x-x-x-1697758654     s3://xxxx/xxxx/xxxrke2-1/on-demand-ip-x-x-x-x-1697758654     13680672 2023-10-19T23:37:34Z
etcd-snapshot-ip-x-x-x-x-1697758682 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697758682        14008352 2023-10-19T23:38:02Z
etcd-snapshot-ip-x-x-x-x-1697758682 s3://xxxx/xxxx/xxxrke2-1/etcd-snapshot-ip-x-x-x-x-1697758682 14008352 2023-10-19T23:38:02Z

Cluster Status:

$ kubectl get nodes
NAME               STATUS   ROLES                       AGE   VERSION
ip-x-x-x-x         Ready    <none>                      69m   v1.28.3+rke2r1
ip-1-1-1-1         Ready    control-plane,etcd,master   74m   v1.28.3+rke2r1
ip-2-2-2-2         Ready    control-plane,etcd,master   71m   v1.28.3+rke2r1
ip-3-3-3-3         Ready    control-plane,etcd,master   70m   v1.28.3+rke2r1

$ kubectl get pods -A
NAMESPACE        NAME                                                   READY   STATUS      RESTARTS        AGE
auto-clusterip   test-clusterip-8496c7779d-h9qrr                        1/1     Running     0               67m
auto-clusterip   test-clusterip-8496c7779d-xz89z                        1/1     Running     0               67m
auto-daemonset   test-daemonset-8j8hb                                   1/1     Running     0               67m
auto-daemonset   test-daemonset-8prn5                                   1/1     Running     0               67m
auto-daemonset   test-daemonset-mrftv                                   1/1     Running     0               67m
auto-daemonset   test-daemonset-x8tpt                                   1/1     Running     0               67m
auto-dns         dnsutils                                               1/1     Running     1 (7m15s ago)   67m
auto-ingress     test-ingress-l5rpr                                     1/1     Running     0               67m
auto-ingress     test-ingress-ttwql                                     1/1     Running     0               67m
auto-nodeport    test-nodeport-644767cc74-8nxvk                         1/1     Running     0               67m
auto-nodeport    test-nodeport-644767cc74-vvkk8                         1/1     Running     0               67m
clusterip        clusterip-pod-demo                                     1/1     Running     0               67m
clusterip        clusterip-pod-demo-2                                   1/1     Running     0               67m
clusterip        clusterip-pod-demo-3                                   1/1     Running     0               67m
kube-system      cloud-controller-manager-ip-1-1-1-1                    1/1     Running     0               75m
kube-system      cloud-controller-manager-ip-2-2-2-2                    1/1     Running     0               73m
kube-system      cloud-controller-manager-ip-3-3-3-3                    1/1     Running     0               71m
kube-system      etcd-ip-1-1-1-1                                        1/1     Running     0               75m
kube-system      etcd-ip-2-2-2-2                                        1/1     Running     0               73m
kube-system      etcd-ip-3-3-3-3                                        1/1     Running     0               71m
kube-system      helm-install-rke2-canal-xg8dz                          0/1     Completed   0               75m
kube-system      helm-install-rke2-coredns-lqn2h                        0/1     Completed   0               75m
kube-system      helm-install-rke2-ingress-nginx-9svb8                  0/1     Completed   0               75m
kube-system      helm-install-rke2-metrics-server-qgkm4                 0/1     Completed   0               75m
kube-system      helm-install-rke2-snapshot-controller-cqbvw            0/1     Completed   2               75m
kube-system      helm-install-rke2-snapshot-controller-crd-dgj9r        0/1     Completed   0               75m
kube-system      helm-install-rke2-snapshot-validation-webhook-qcnml    0/1     Completed   0               75m
kube-system      kube-apiserver-ip-1-1-1-1                              1/1     Running     0               75m
kube-system      kube-apiserver-ip-2-2-2-2                              1/1     Running     0               73m
kube-system      kube-apiserver-ip-3-3-3-3                              1/1     Running     0               71m
kube-system      kube-controller-manager-ip-1-1-1-1                     1/1     Running     0               75m
kube-system      kube-controller-manager-ip-2-2-2-2                     1/1     Running     0               73m
kube-system      kube-controller-manager-ip-3-3-3-3                     1/1     Running     0               71m
kube-system      kube-proxy-ip-172-31-20-155                            1/1     Running     0               71m
kube-system      kube-proxy-ip-1-1-1-1                                  1/1     Running     0               75m
kube-system      kube-proxy-ip-2-2-2-2                                  1/1     Running     0               73m
kube-system      kube-proxy-ip-3-3-3-3                                  1/1     Running     0               72m
kube-system      kube-scheduler-ip-1-1-1-1                              1/1     Running     0               75m
kube-system      kube-scheduler-ip-2-2-2-2                              1/1     Running     0               73m
kube-system      kube-scheduler-ip-3-3-3-3                              1/1     Running     0               71m
kube-system      rke2-canal-2h8gn                                       2/2     Running     0               74m
kube-system      rke2-canal-j6btm                                       2/2     Running     0               71m
kube-system      rke2-canal-qfmh4                                       2/2     Running     0               72m
kube-system      rke2-canal-vrd8j                                       2/2     Running     0               73m
kube-system      rke2-coredns-rke2-coredns-6b795db654-6rs4h             1/1     Running     0               73m
kube-system      rke2-coredns-rke2-coredns-6b795db654-ljzq2             1/1     Running     0               74m
kube-system      rke2-coredns-rke2-coredns-autoscaler-945fbd459-z9vpk   1/1     Running     0               74m
kube-system      rke2-ingress-nginx-controller-jqkqm                    1/1     Running     0               70m
kube-system      rke2-ingress-nginx-controller-knkvf                    1/1     Running     0               71m
kube-system      rke2-ingress-nginx-controller-qbzqz                    1/1     Running     0               73m
kube-system      rke2-ingress-nginx-controller-srr42                    1/1     Running     0               73m
kube-system      rke2-metrics-server-544c8c66fc-xrhqw                   1/1     Running     0               73m
kube-system      rke2-snapshot-controller-59cc9cd8f4-47ljg              1/1     Running     0               72m
kube-system      rke2-snapshot-validation-webhook-54c5989b65-m2xg7      1/1     Running     0               73m

$ sudo systemctl status rke2-server
rke2-server.service - Rancher Kubernetes Engine v2 (server)
     Loaded: loaded (/usr/local/lib/systemd/system/rke2-server.service; enabled; vendor preset: enabled)
     Active: active (running) since Thu 2023-10-19 23:30:03 UTC; 1h 21min ago
       Docs: https://github.com/rancher/rke2#readme
    Process: 1079206 ExecStartPre=/bin/sh -xc ! /usr/bin/systemctl is-enabled --quiet nm-cloud-setup.service (code=exited, status=0/SUCCESS)
    Process: 1079208 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 1079209 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 1079210 (rke2)
      Tasks: 198
     Memory: 2.0G
        CPU: 7min 49.329s
     CGroup: /system.slice/rke2-server.service
             ├─1077910 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id b87477474d32182479309c80ad5c399ad20bca9618e2480245ff7208dc8344de -address /run/k3s/containerd/containerd.sock
             ├─1077992 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id a31d78e2b54230d34832e27249e933e711d608f608a02c695f76174ded786c99 -address /run/k3s/containerd/containerd.sock
             ├─1078086 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id e4cdc57698d13ee9cb9c0d07e1f7133a39fb1b2bbff2c5d34394900002b3f756 -address /run/k3s/containerd/containerd.sock
             ├─1078112 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id bdd026d1568f4245ec5390a55429dbd15f5f5297edac468302ff1f82b7df5ee1 -address /run/k3s/containerd/containerd.sock
             ├─1078244 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id 2b691fb98ef5110e7a0b299e35e4e5c6ef96de30fa3b20b96b586d312099897a -address /run/k3s/containerd/containerd.sock
             ├─1078960 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id ff286cf90014c2716dc3f834c1dada0dc43b1c77b0623afa773a452086d9b567 -address /run/k3s/containerd/containerd.sock
             ├─1079210 "/usr/local/bin/rke2 server"
             ├─1079219 containerd -c /var/lib/rancher/rke2/agent/etc/containerd/config.toml -a /run/k3s/containerd/containerd.sock --state /run/k3s/containerd --root /var/lib/rancher/rke2/agent/containerd
             ├─1079324 kubelet --volume-plugin-dir=/var/lib/kubelet/volumeplugins --file-check-frequency=5s --sync-frequency=30s --address=0.0.0.0 --anonymous-auth=false --authentication-token-webhook=true --authorization-mode=Webhook --cgroup-driver=systemd --client-ca-file=/var/lib/rancher/r>
             ├─1079613 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id aab54933c210bf79738ab0df6473e79fd490e5135d96e4214da384b12259340d -address /run/k3s/containerd/containerd.sock
             ├─1081368 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id ab399aa7171d23d8715c4202ce03b9d9b41a050201a11a3ed603f8c1920e1673 -address /run/k3s/containerd/containerd.sock
             ├─1082149 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id 0aa5f31542c9732afa91b1b6704395e4a655f8340ed2788f79bb8cbc7d83b6bb -address /run/k3s/containerd/containerd.sock
             ├─1082934 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id 9721594628fad3164ab3f167bc32c5df471a535c7ecf9939efe385401bf34555 -address /run/k3s/containerd/containerd.sock
             ├─1083655 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id 6e7c09b78abf27760e76bd1a8451dc3a6b25c211104d8ce772b90e4a6afbf6b8 -address /run/k3s/containerd/containerd.sock
             ├─1083676 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id 34e5aa9aae15848b2b7719f8d1368d65f59edde0fccdb673f60fdd786bc33b82 -address /run/k3s/containerd/containerd.sock
             └─1091795 /var/lib/rancher/rke2/data/v1.28.3-dev.0d0d0e48-9e5c5ea79a97/bin/containerd-shim-runc-v2 -namespace k8s.io -id 40ba6e06fb997709d5488ac3fc1ab5eb2cefa0ed17e8262986c4f7ff900749dd -address /run/k3s/containerd/containerd.sock

Oct 20 00:51:01 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:01Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697762762"
Oct 20 00:51:01 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:01Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697762763"
Oct 20 00:51:01 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:01Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697762824"
Oct 20 00:51:01 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:01Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697762883"
Oct 20 00:51:01 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:01Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697762943"
Oct 20 00:51:02 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:02Z" level=info msg="Deleting ETCDSnapshotFile for etcd-snapshot-ip-2-2-2-2-1697763004"
Oct 20 00:51:02 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:02Z" level=info msg="Creating ETCDSnapshotFile for on-demand-ip-1-1-1-1-1697758654"
Oct 20 00:51:02 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:02Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697763003"
Oct 20 00:51:03 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:03Z" level=info msg="Creating ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697762824"
Oct 20 00:51:06 ip-1-1-1-1 rke2[1079210]: time="2023-10-20T00:51:06Z" level=info msg="Reconciliation of ETCDSnapshotFile resources complete"

@brandond
Copy link
Contributor

brandond commented Oct 20, 2023 •
edited
Loading

@aganesh-suse you need to put data in the extra-metadata configmap. This does not exist by default, but Rancher uses it to store metadata about the cluster that is stored alongside the snapshots. It is this metadata that usually consumes space in the configmap.

Here is an example that will fill a configmap very quickly, as another copy of the entire text (in base64 encoding) will be stored in the snapshot configmap for each snapshot taken.

apiVersion: v1
kind: ConfigMap
metadata:
  name: rke2-etcd-snapshot-extra-metadata
  namespace: kube-system
data:
  lorem_ipsum: |-
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.
    Lorem ipsum dolor sit amet, consectetur adipiscing elit. Vestibulum bibendum, quam ut imperdiet efficitur, est justo semper diam, ut elementum tellus dolor quis dolor. Nunc euismod sem in dolor volutpat, id feugiat arcu molestie. Vivamus id laoreet mauris, sit amet consequat ligula. Nunc eu mi vitae orci finibus elementum at sed magna. Nam porta venenatis diam quis scelerisque. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Nunc quis mauris enim. Suspendisse turpis ligula, varius condimentum eleifend porttitor, eleifend eget sapien. Phasellus et felis eget lacus rhoncus porta a faucibus magna. Phasellus vel nisi est. Etiam condimentum felis quam, vitae tempor enim suscipit fermentum. Ut fringilla semper lorem, non auctor purus convallis eget.
    Suspendisse in dolor ut diam eleifend porta eu quis massa. Fusce ultrices, magna quis varius mattis, dolor lorem porttitor sapien, quis pharetra dui neque ut velit. Nunc gravida quam sit amet lacus pulvinar, sit amet fermentum nulla rutrum. Maecenas feugiat vestibulum ipsum. Vivamus pellentesque hendrerit purus non laoreet. Cras vitae dolor orci. Nullam malesuada diam ut risus sollicitudin egestas. Morbi pharetra, purus id gravida cursus, erat neque congue leo, eu hendrerit arcu nibh ac ante. Fusce urna velit, placerat sit amet ipsum non, rhoncus fermentum libero. Quisque fringilla tincidunt lectus, ac pretium urna elementum et. Vestibulum elementum, nulla eget mollis ultrices, neque quam semper ex, quis rhoncus nulla libero vitae elit.
    Praesent blandit ligula in diam sagittis suscipit. Nunc id eleifend sem. Nunc neque nulla, congue quis efficitur a, sollicitudin non odio. Vivamus vulputate nunc justo, id dictum odio tempor a. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Etiam facilisis, massa in ultricies feugiat, enim lorem consequat nibh, eu tempor est purus volutpat nisl. Maecenas viverra dui vitae leo ultrices, vel tristique lectus porttitor. Nulla scelerisque nisi leo, vel tincidunt nisi vehicula ut. Nam hendrerit auctor ipsum, id volutpat dui vehicula sit amet. Nulla imperdiet nulla nunc, eu efficitur lorem vehicula quis. Morbi augue eros, pulvinar at turpis et, dapibus facilisis orci. Lorem ipsum dolor sit amet, consectetur adipiscing elit. Curabitur arcu erat, auctor vitae lorem rutrum, porttitor iaculis neque.
    Aenean tempus, risus in hendrerit convallis, nisi nisl tincidunt purus, eu vehicula ligula ante eget eros. Duis pulvinar nulla a magna congue suscipit. Vivamus at nulla sodales, elementum sapien a, bibendum lectus. Vestibulum fermentum sem ut ex congue, varius tempor nisi placerat. Cras dapibus fermentum mollis. Nunc ut iaculis dolor, volutpat tincidunt purus. Sed volutpat ac dolor a elementum. Curabitur tincidunt rhoncus neque, eu ultricies mauris sagittis in. Duis pulvinar mattis sodales. Ut aliquam, mauris a mollis faucibus, lectus ligula fringilla lacus, in interdum nibh massa a turpis. Mauris at iaculis ligula. Nunc gravida enim in urna egestas, et placerat odio fermentum. Phasellus sagittis ipsum nisi, id sagittis est mollis ac. Sed id justo massa.
    Mauris tincidunt quis libero vitae varius. Ut vel dolor sed libero aliquam suscipit. Cras augue quam, suscipit vel porta non, venenatis ac neque. Nullam sit amet lectus eu justo tincidunt venenatis. Nunc sit amet tellus nulla. Nulla sed aliquet diam, ac sagittis urna. Integer auctor feugiat turpis luctus porta. Vestibulum ante ipsum primis in faucibus orci luctus et ultrices posuere cubilia curae; Morbi feugiat ac nulla placerat ornare. Proin hendrerit convallis dolor, hendrerit congue arcu viverra sed. Nam tempor id arcu quis mollis.

@aganesh-suse
Copy link

aganesh-suse commented Oct 20, 2023 •
edited
Loading

Thanks a lot @brandond for the yaml/metadata example you provided.
Was able to reproduce with the same pretty easily!! I did not even have to set a retention value over 5 👍

Reproduced Setup Output:

$ rke2 -v
rke2 version v1.28.2+rke2r1 (7466261e4792e68baa2cc0c2afd3dcc929d72061)
go version go1.20.8 X:boringcrypto

$ kubectl get cm -n kube-system | grep etcd
rke2-etcd-snapshot-extra-metadata                      1      50m
rke2-etcd-snapshots                                    25     61m

$ sudo rke2 etcd-snapshot save
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Managed etcd cluster bootstrap already complete and initialized
INFO[0000] Applying CRD helmcharts.helm.cattle.io
INFO[0000] Applying CRD helmchartconfigs.helm.cattle.io
INFO[0000] Applying CRD addons.k3s.cattle.io
INFO[0000] Creating rke2-supervisor event broadcaster
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-2-2-2-2-1697833861
{"level":"info","ts":"2023-10-20T20:31:01.086746Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-2-2-2-2-1697833861.part"}
{"level":"info","ts":"2023-10-20T20:31:01.094681Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-20T20:31:01.094781Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-20T20:31:02.81422Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-20T20:31:02.854029Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"30 MB","took":"1 second ago"}
{"level":"info","ts":"2023-10-20T20:31:02.854265Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-2-2-2-2-1697833861"}
FATA[0002] failed to save local snapshot data to configmap: ConfigMap "rke2-etcd-snapshots" is invalid: []: Too long: must have at most 1048576 bytes

$ sudo rke2 etcd-snapshot list
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Checking if S3 bucket xxxx exists
INFO[0000] S3 bucket xxxx exists
Name                                     Size     Created
etcd-snapshot-ip-2-2-2-2-1697832301 10690592 2023-10-20T20:05:02Z
etcd-snapshot-ip-2-2-2-2-1697832360 10924064 2023-10-20T20:06:01Z
etcd-snapshot-ip-2-2-2-2-1697832421 12152864 2023-10-20T20:07:03Z
etcd-snapshot-ip-2-2-2-2-1697832481 12152864 2023-10-20T20:08:03Z
etcd-snapshot-ip-2-2-2-2-1697832604 29290528 2023-10-20T20:10:07Z
on-demand-ip-2-2-2-2-1697832507     16314400 2023-10-20T20:08:28Z
on-demand-ip-2-2-2-2-1697832510     17616928 2023-10-20T20:08:32Z
on-demand-ip-2-2-2-2-1697832514     20803616 2023-10-20T20:08:36Z

Validation Setup Output:

$ rke2 -v
rke2 version v1.28.3-rc2+rke2r1 (0d0d0e4879fdf95254461e3a49224f75d7b2dc3d)
go version go1.20.10 X:boringcrypto

$ kubectl get cm -n kube-system | grep etcd
rke2-etcd-snapshot-extra-metadata                      1      163m
rke2-etcd-snapshots                                    7      168m

$ sudo rke2 etcd-snapshot save
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Saving etcd snapshot to /var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697834392
{"level":"info","ts":"2023-10-20T20:39:51.567692Z","caller":"snapshot/v3_snapshot.go:65","msg":"created temporary db file","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697834392.part"}
{"level":"info","ts":"2023-10-20T20:39:51.572776Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:212","msg":"opened snapshot stream; downloading"}
{"level":"info","ts":"2023-10-20T20:39:51.572873Z","caller":"snapshot/v3_snapshot.go:73","msg":"fetching snapshot","endpoint":"https://127.0.0.1:2379"}
{"level":"info","ts":"2023-10-20T20:39:55.829453Z","logger":"client","caller":"v3@v3.5.9-k3s1/maintenance.go:220","msg":"completed snapshot read; closing"}
{"level":"info","ts":"2023-10-20T20:39:56.042294Z","caller":"snapshot/v3_snapshot.go:88","msg":"fetched snapshot","endpoint":"https://127.0.0.1:2379","size":"236 MB","took":"4 seconds ago"}
{"level":"info","ts":"2023-10-20T20:39:56.042401Z","caller":"snapshot/v3_snapshot.go:97","msg":"saved","path":"/var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697834392"}
INFO[0004] Saving snapshot metadata to /var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-x-x-x-x-1697834392
INFO[0004] Checking if S3 bucket xxxx exists
INFO[0004] S3 bucket xxxx exists
INFO[0004] Saving etcd snapshot on-demand-ip-x-x-x-x-1697834392 to S3
INFO[0004] Uploading snapshot to s3://xxxx//var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697834392
INFO[0008] Uploaded snapshot metadata s3://xxxx//var/lib/rancher/rke2/server/db/.metadata/on-demand-ip-x-x-x-x-1697834392
INFO[0008] S3 upload complete for on-demand-ip-x-x-x-x-1697834392
INFO[0008] Reconciling ETCDSnapshotFile resources
INFO[0008] Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697834105
INFO[0008] Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697834160
INFO[0008] Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697834225
INFO[0008] Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697834285
INFO[0008] Deleting ETCDSnapshotFile for etcd-snapshot-ip-1-1-1-1-1697834345
INFO[0008] Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697834283
INFO[0008] Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697834340
INFO[0009] Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697834165
INFO[0009] Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697834224
INFO[0009] Creating ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697824612
INFO[0009] Creating ETCDSnapshotFile for on-demand-ip-x-x-x-x-1697824616
INFO[0010] Creating ETCDSnapshotFile for etcd-snapshot-ip-x-x-x-x-1697834102
INFO[0010] Reconciliation of ETCDSnapshotFile resources complete

$ sudo rke2 etcd-snapshot list
WARN[0000] Unknown flag --token found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-retention found in config.yaml, skipping
WARN[0000] Unknown flag --etcd-snapshot-schedule-cron found in config.yaml, skipping
WARN[0000] Unknown flag --write-kubeconfig-mode found in config.yaml, skipping
WARN[0000] Unknown flag --node-external-ip found in config.yaml, skipping
INFO[0000] Checking if S3 bucket xxxx exists
INFO[0000] S3 bucket xxxx exists
Name                                      Location                                                                                          Size      Created
on-demand-ip-x-x-x-x-1697824612     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697824612            35487776  2023-10-20T17:56:52Z
on-demand-ip-x-x-x-x-1697824612     s3://xxxx/rke2-1/on-demand-ip-x-x-x-x-1697824612                                            35487776  2023-10-20T17:56:52Z
on-demand-ip-x-x-x-x-1697824616     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697824616            35487776  2023-10-20T17:56:56Z
on-demand-ip-x-x-x-x-1697824616     s3://xxxx/rke2-1/on-demand-ip-x-x-x-x-1697824616                                            35487776  2023-10-20T17:56:56Z
on-demand-ip-x-x-x-x-1697834392     file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697834392            236027936 2023-10-20T20:39:52Z
on-demand-ip-x-x-x-x-1697834392     s3://xxxx/rke2-1/on-demand-ip-x-x-x-x-1697834392                                            236027936 2023-10-20T20:39:52Z
etcd-snapshot-ip-x-x-x-x-1697834404 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697834404        236027936 2023-10-20T20:40:04Z
etcd-snapshot-ip-x-x-x-x-1697834404 s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697834404                                        236027936 2023-10-20T20:40:04Z
etcd-snapshot-ip-x-x-x-x-1697834462 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697834462        237781024 2023-10-20T20:41:02Z
etcd-snapshot-ip-x-x-x-x-1697834462 s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697834462                                        237781024 2023-10-20T20:41:02Z
etcd-snapshot-ip-x-x-x-x-1697834522 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697834522        241586208 2023-10-20T20:42:02Z
etcd-snapshot-ip-x-x-x-x-1697834522 s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697834522                                        241586208 2023-10-20T20:42:02Z
etcd-snapshot-ip-x-x-x-x-1697834581 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697834581        241586208 2023-10-20T20:43:01Z
etcd-snapshot-ip-x-x-x-x-1697834581 s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697834581                                        241586208 2023-10-20T20:43:01Z
etcd-snapshot-ip-x-x-x-x-1697834642 file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697834642        241586208 2023-10-20T20:44:02Z
etcd-snapshot-ip-x-x-x-x-1697834642 s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697834642                                        241586208 2023-10-20T20:44:02Z

$ kubectl get etcdsnapshotfile
NAME                                                     SNAPSHOTNAME                                NODE               LOCATION                                                                                            SIZE        CREATIONTIME
local-etcd-snapshot-ip-1-1-1-1-1697835542-444cbe    etcd-snapshot-ip-1-1-1-1-1697835542    ip-1-1-1-1    file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1-1-1-1-1697835542           241811488   2023-10-20T20:59:02Z
local-etcd-snapshot-ip-1-1-1-1-1697835604-55adf0    etcd-snapshot-ip-1-1-1-1-1697835604    ip-1-1-1-1    file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1-1-1-1-1697835604           241811488   2023-10-20T21:00:04Z
local-etcd-snapshot-ip-1-1-1-1-1697835663-ea05bb    etcd-snapshot-ip-1-1-1-1-1697835663    ip-1-1-1-1    file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1-1-1-1-1697835663           241811488   2023-10-20T21:01:03Z
local-etcd-snapshot-ip-1-1-1-1-1697835722-526db8    etcd-snapshot-ip-1-1-1-1-1697835722    ip-1-1-1-1    file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1-1-1-1-1697835722           241811488   2023-10-20T21:02:02Z
local-etcd-snapshot-ip-1-1-1-1-1697835781-beb266    etcd-snapshot-ip-1-1-1-1-1697835781    ip-1-1-1-1    file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-1-1-1-1-1697835781           241811488   2023-10-20T21:03:01Z
local-etcd-snapshot-ip-x-x-x-x-1697835544-c3a4c4   etcd-snapshot-ip-x-x-x-x-1697835544   ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697835544          241586208   2023-10-20T20:59:04Z
local-etcd-snapshot-ip-x-x-x-x-1697835601-ce868f   etcd-snapshot-ip-x-x-x-x-1697835601   ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697835601          241586208   2023-10-20T21:00:01Z
local-etcd-snapshot-ip-x-x-x-x-1697835665-90a208   etcd-snapshot-ip-x-x-x-x-1697835665   ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697835665          241586208   2023-10-20T21:01:05Z
local-etcd-snapshot-ip-x-x-x-x-1697835723-b369a0   etcd-snapshot-ip-x-x-x-x-1697835723   ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697835723          241586208   2023-10-20T21:02:03Z
local-etcd-snapshot-ip-x-x-x-x-1697835782-e8f546   etcd-snapshot-ip-x-x-x-x-1697835782   ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-x-x-x-x-1697835782          241586208   2023-10-20T21:03:02Z
local-etcd-snapshot-ip-2-2-2-2-1697835541-77e14e   etcd-snapshot-ip-2-2-2-2-1697835541   ip-2-2-2-2   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-2-2-2-2-1697835541          242982944   2023-10-20T20:59:01Z
local-etcd-snapshot-ip-2-2-2-2-1697835601-c9f642   etcd-snapshot-ip-2-2-2-2-1697835601   ip-2-2-2-2   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-2-2-2-2-1697835601          242982944   2023-10-20T21:00:01Z
local-etcd-snapshot-ip-2-2-2-2-1697835664-1c60dc   etcd-snapshot-ip-2-2-2-2-1697835664   ip-2-2-2-2   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-2-2-2-2-1697835664          242982944   2023-10-20T21:01:04Z
local-etcd-snapshot-ip-2-2-2-2-1697835720-f31442   etcd-snapshot-ip-2-2-2-2-1697835720   ip-2-2-2-2   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-2-2-2-2-1697835720          242982944   2023-10-20T21:02:00Z
local-etcd-snapshot-ip-2-2-2-2-1697835785-5d5b85   etcd-snapshot-ip-2-2-2-2-1697835785   ip-2-2-2-2   file:///var/lib/rancher/rke2/server/db/snapshots/etcd-snapshot-ip-2-2-2-2-1697835785          242982944   2023-10-20T21:03:05Z
local-on-demand-ip-x-x-x-x-1697824612-479e13       on-demand-ip-x-x-x-x-1697824612       ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697824612              35487776    2023-10-20T17:56:52Z
local-on-demand-ip-x-x-x-x-1697824616-707129       on-demand-ip-x-x-x-x-1697824616       ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697824616              35487776    2023-10-20T17:56:56Z
local-on-demand-ip-x-x-x-x-1697834392-430c7b       on-demand-ip-x-x-x-x-1697834392       ip-x-x-x-x   file:///var/lib/rancher/rke2/server/db/snapshots/on-demand-ip-x-x-x-x-1697834392              236027936   2023-10-20T20:39:52Z
s3-etcd-snapshot-ip-x-x-x-x-1697835544-28276c      etcd-snapshot-ip-x-x-x-x-1697835544   ip-x-x-x-x   s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697835544   241586208   2023-10-20T20:59:04Z
s3-etcd-snapshot-ip-x-x-x-x-1697835601-056696      etcd-snapshot-ip-x-x-x-x-1697835601   ip-x-x-x-x   s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697835601   241586208   2023-10-20T21:00:01Z
s3-etcd-snapshot-ip-x-x-x-x-1697835665-c0ba71      etcd-snapshot-ip-x-x-x-x-1697835665   ip-x-x-x-x   s3://xxxx/rke2-1/etcd-snapshot-ip-x-x-x-x-1697835665   241586208   2023-10-20T21:01:05Z
s3-etcd-snapshot-ip-2-2-2-2-1697835541-ab7c2d      etcd-snapshot-ip-2-2-2-2-1697835541   ip-2-2-2-2   s3://xxxx/rke2-3/etcd-snapshot-ip-2-2-2-2-1697835541   242982944   2023-10-20T20:59:01Z
s3-etcd-snapshot-ip-2-2-2-2-1697835601-049da2      etcd-snapshot-ip-2-2-2-2-1697835601   ip-2-2-2-2   s3://xxxx/rke2-3/etcd-snapshot-ip-2-2-2-2-1697835601   242982944   2023-10-20T21:00:01Z
s3-etcd-snapshot-ip-2-2-2-2-1697835664-2202b7      etcd-snapshot-ip-2-2-2-2-1697835664   ip-2-2-2-2   s3://xxxx/rke2-3/etcd-snapshot-ip-2-2-2-2-1697835664   242982944   2023-10-20T21:01:04Z
s3-etcd-snapshot-ip-2-2-2-2-1697835720-71415d      etcd-snapshot-ip-2-2-2-2-1697835720   ip-2-2-2-2   s3://xxxx/rke2-3/etcd-snapshot-ip-2-2-2-2-1697835720   242982944   2023-10-20T21:02:00Z
s3-on-demand-ip-x-x-x-x-1697824612-2af282          on-demand-ip-x-x-x-x-1697824612       ip-x-x-x-x   s3://xxxx/rke2-1/on-demand-ip-x-x-x-x-1697824612       35487776    2023-10-20T17:56:52Z
s3-on-demand-ip-x-x-x-x-1697824616-2ecb29          on-demand-ip-x-x-x-x-1697824616       ip-x-x-x-x   s3://xxxx/rke2-1/on-demand-ip-x-x-x-x-1697824616       35487776    2023-10-20T17:56:56Z
s3-on-demand-ip-x-x-x-x-1697834392-ab1326          on-demand-ip-x-x-x-x-1697834392       ip-x-x-x-x   s3://xxxx/rke2-1/on-demand-ip-x-x-x-x-1697834392       236027936   2023-10-20T20:39:52Z

@wernerfred
Copy link

wernerfred commented Oct 31, 2023 •
edited
Loading

Hi @aganesh-suse can you tell me why you closed this? We run into the same issue but following this discussion i couldn't find a solution or am i missing something?

Found it in #4883

User Facing Change: RKE2 now tracks snapshots using custom resource definitions. This resolves an issue where the configmap previously used to track snapshot metadata could grow excessively large and fail to update when new snapshots were taken.

@Oats87 Oats87 mentioned this issue Feb 14, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@brandond brandond

@aganesh-suse aganesh-suse

Labels
kind/bug Something isn't working
Projects
None yet
Milestone
v1.28.3+rke2r1
Development

No branches or pull requests