S3 Snapshot Retention policy is too aggressive

[Oats87]

Environmental Info:
RKE2 Version: v1.25.15+rke2r2, v1.27.7+rke2r2

Node(s) CPU architecture, OS, and Version:
N/A

Cluster Configuration:
Multi-master

Describe the bug:
According to the r/r issue, RKE2 is too aggressive at pruning S3 snapshots.

Steps To Reproduce:
See Rancher issue.

Additional context / logs:
rancher/rancher#43872

[brandond]

That is correct - the S3 retention count applies to ALL snapshots stored on S3, not to the number of snapshots per node.

On earlier releases of RKE2, snapshots were pruned from S3 only by the node that uploaded them, but this lead to snapshots never being pruned when their source node was deleted or otherwise removed from the cluster. For clusters where nodes are upgraded or patched by replacement, this lead to S3 snapshot counts growing without bound due to the orphaned snapshots never being cleaned up.

One option would be to provide separate settings and commands for pruning snapshots from nodes that are no longer cluster members, but this would probably have to go through the RFE process.

[Oats87]

OK, so it sounds like default behavior was changed and this is affecting our users as it was not clearly communicated through to them. Perhaps we need to look at our process to see how to avoid this in the future.

cc: @snasovich @cwayne18

[brandond]

It was changed based on complaints from users.

Ref:

• etcd snapshot cleanup fails if node name changes #3714
• Fixed the etcd retention to delete orphaned snapshots based on the date k3s-io/k3s#8177

[thomashoell]

That is correct - the S3 retention count applies to ALL snapshots stored on S3, not to the number of snapshots per node.

On earlier releases of RKE2, snapshots were pruned from S3 only by the node that uploaded them, but this lead to snapshots never being pruned when their source node was deleted or otherwise removed from the cluster. For clusters where nodes are upgraded or patched by replacement, this lead to S3 snapshot counts growing without bound due to the orphaned snapshots never being cleaned up.

One option would be to provide separate settings and commands for pruning snapshots from nodes that are no longer cluster members, but this would probably have to go through the RFE process.

But right now the behaviour is not consistent with what the UI says and also what you'd expect.
I would be totally fine with ONE consistent snapshot per snapshot cycle in S3, but right now RKE2 removes anything except ${num_master_nodes}*${snapshot_retention_count}. So if I have 3 master nodes and a retention count of 5, I'm left with 3 snapshots of the latest cycle and 2 snapshots of the 2nd latest.

We've worked around by increasing the retention count from 5 to 15. This will lead to 15 snapshots per master node but at least we have 5 usable snapshots in S3 instead of 2...

[brandond]

right now the behaviour is not consistent with what the UI says

That would be the rancher UI, not the RKE2 UI. It sounds like the rancher UI text needs to be updated.

RKE2 removes anything except ${num_master_nodes}*${snapshot_retention_count}

That is not correct. RKE2 stores snapshot_retention_count snapshots locally on each node, and snapshot_retention_count snapshots on s3. So your total COMBINED snapshot count, across both nodes and s3, would would be (num_master_nodes * snapshot_retention_count) + snapshot_retention_count

If you are looking at just s3, then yes you would probably want to align your retention count so that you keep the desired number of snapshots on s3 - if you have 3 nodes and want 2 cycles worth of snapshots, then you should set retention to 6, for example.

We will discuss how to enhance this to provide more granular control over snapshot retention, but it will probably require splitting the settings to allow separate counts for local and s3.

[horihel]

We will discuss how to enhance this to provide more granular control over snapshot retention, but it will probably require splitting the settings to allow separate counts for local and s3.

splitting the settings is probably the most sensible approach.

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[brandond]

Unstale

[fayak]

I'm not sure this is exactly related, but even I set the option Keep the last X to some value like 15, I still have 1 snapshot of each of my 5 nodes, not 15 per nodes nor 15 in total (3 per nodes)

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[fayak]

This is still relevant. S3 backups are messed up and don't work as expected

[github-actions]

This repository uses a bot to automatically label issues which have not had any activity (commit/comment/label) for 45 days. This helps us manage the community issues better. If the issue is still relevant, please add a comment to the issue so the bot can remove the label and we know it is still valid. If it is no longer relevant (or possibly fixed in the latest release), the bot will automatically close the issue in 14 days. Thank you for your contributions.

[horihel]

still very much an issue - workaround is to enable versioning and/or soft-delete on your s3 storage. Not sure if rke2 registers recovered backups automatically though.