Migrate to using custom resource to store etcd snapshot metadata

docs/adrs/etcd-snapshot-cr.md

@@ -0,0 +1,60 @@
+# Store etcd snapshot metadata in a Custom Resource
+
+Date: 2023-07-27
+
+## Status
+
+Accepted
+
+## Context
+
+K3s currently stores a list of etcd snapshots and associated metadata in a ConfigMap. Other downstream
+projects and controllers consume the content of this ConfigMap in order to present cluster administrators with
+a list of snapshots that can be restored.
+
+On clusters with more than a handful of nodes, and reasonable snapshot intervals and retention periods, the snapshot
+list ConfigMap frequently reaches the maximum size allowed by Kubernetes, and fails to store any additional information.
+The snapshots are still created, but they cannot be discovered by users or accessed by tools that consume information
+from the ConfigMap.
+
+When this occurs, the K3s service log shows errors such as:
+```
+level=error msg="failed to save local snapshot data to configmap: ConfigMap \"k3s-etcd-snapshots\" is invalid: []: Too long: must have at most 1048576 bytes"
+```
+
+A side-effect of this is that snapshot metadata is lost if the ConfigMap cannot be updated, as the list is the only place that it is stored.
+
+Reference:
+* https://github.com/rancher/rke2/issues/4495
+* https://github.com/k3s-io/k3s/blob/36645e7311e9bdbbf2adb79ecd8bd68556bc86f6/pkg/etcd/etcd.go#L1503-L1516
+
+### Existing Work
+
+Rancher already has a `rke.cattle.io/v1 ETCDSnapshot` Custom Resource that contains the same information after it's been
+imported by the management cluster:
+* https://github.com/rancher/rancher/blob/027246f77f03b82660dc2e91df6bf2cd549163f0/pkg/apis/rke.cattle.io/v1/etcd.go#L48-L74
+
+It is unlikely that we would want to use this custom resource in its current package; we may be able to negotiate moving
+it into a neutral project for use by both projects.
+
+## Decision
+
+1. Instead of populating snapshots into a ConfigMap using the JSON serialization of the private `snapshotFile` type, K3s
+   will manage creation of an new Custom Resource Definition with similar fields.
+2. Metadata on each snapshot will be stored in a distinct Custom Resource.
+3. The new Custom Resource will be cluster-scoped, as etcd and its snapshots are a cluster-level resource.
+4. Snapshot metadata will also be written alongside snapshot files created on disk and/or uploaded to S3. The metadata
+   files will have the same basename as their corresponding snapshot file.
+5. A hash of the server token will be stored as an annotation on the Custom Resource, and stored as metadata on snapshots uploaded to S3.
+   This hash should be compared to a current etcd snapshot's token hash to determine if the server token must be rolled back as part of the
+   snapshot restore process.
+6. Downstream consumers of etcd snapshot lists will migrate to watching Custom Resource types, instead of the ConfigMap.
+7. K3s will observe a three minor version transition period, where both the new Custom Resources, and the existing
+   ConfigMap, will both be used.
+8. During the transition period, older snapshot metadata may be removed from the ConfigMap while those snapshots still
+   exist and are referenced by new Custom Resources, if the ConfigMap exceeds a preset size or key count limit.
+
+## Consequences
+
+* Snapshot metadata will no longer be lost when the number of snapshots exceeds what can be stored in the ConfigMap.
+* There will be some additional complexity in managing the new Custom Resource, and working with other projects to migrate to using it.

hack/crdgen.go

@@ -0,0 +1,13 @@
+package main
+
+import (
+	"os"
+
+	k3scrd "github.com/k3s-io/k3s/pkg/crd"
+	_ "github.com/k3s-io/k3s/pkg/generated/controllers/k3s.cattle.io/v1"
+	"github.com/rancher/wrangler/pkg/crd"
+)
+
+func main() {
+	crd.Print(os.Stdout, k3scrd.List())
+}

pkg/apis/k3s.cattle.io/v1/types.go

@@ -1,20 +1,105 @@
 package v1
 
 import (
+	"k8s.io/apimachinery/pkg/api/resource"
 	metav1 "k8s.io/apimachinery/pkg/apis/meta/v1"
 )
 
 // +genclient
+// +genclient:noStatus
 // +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
 
+// Addon is used to track application of a manifest file on disk. It mostly exists so that the wrangler DesiredSet
+// Apply controller has an object to track as the owner, and ensure that all created resources are tracked when the
+// manifest is modified or removed.
 type Addon struct {
 	metav1.TypeMeta   `json:",inline"`
 	metav1.ObjectMeta `json:"metadata,omitempty"`
 
+	// Spec provides information about the on-disk manifest backing this resource.
 	Spec AddonSpec `json:"spec,omitempty"`
 }
 
 type AddonSpec struct {
-	Source   string `json:"source,omitempty"`
-	Checksum string `json:"checksum,omitempty"`
+	// Source is the Path on disk to the manifest file that this Addon tracks.
+	Source string `json:"source,omitempty" column:""`
+	// Checksum is the SHA256 checksum of the most recently successfully applied manifest file.
+	Checksum string `json:"checksum,omitempty" column:""`
+}
+
+// +genclient
+// +genclient:nonNamespaced
+// +k8s:deepcopy-gen:interfaces=k8s.io/apimachinery/pkg/runtime.Object
+
+// ETCDSnapshot tracks a point-in-time snapshot of the etcd datastore.
+type ETCDSnapshotFile struct {
+	metav1.TypeMeta   `json:",inline"`
+	metav1.ObjectMeta `json:"metadata,omitempty"`
+
+	// Spec defines properties of an etcd snapshot file
+	Spec ETCDSnapshotSpec `json:"spec,omitempty"`
+	// Status represents current information about a snapshot.
+	Status ETCDSnapshotStatus `json:"status,omitempty"`
+}
+
+// ETCDSnapshotSpec desribes an etcd snapshot file
+type ETCDSnapshotSpec struct {
+	// SnapshotName contains the base name of the snapshot file. CLI actions that act
+	// on snapshots stored locally or within a pre-configured S3 bucket and
+	// prefix usually take the snapshot name as their argument.
+	SnapshotName string `json:"snapshotName" column:""`
+	// NodeName contains the name of the node that took the snapshot.
+	NodeName string `json:"nodeName" column:"name=Node"`
+	// Location is the absolute file:// or s3:// URI address of the snapshot.
+	Location string `json:"location" column:""`
+	// Metadata contains point-in-time snapshot of the contents of the
+	// k3s-etcd-snapshot-extra-metadata ConfigMap's data field, at the time the
+	// snapshot was taken. This is intended to contain data about cluster state
+	// that may be important for an external system to have available when restoring
+	// the snapshot.
+	Metadata map[string]string `json:"metadata,omitempty"`
+	// S3 contains extra metadata about the S3 storage system holding the
+	// snapshot. This is guaranteed to be set for all snapshots uploaded to S3.
+	// If not specified, the snapshot was not uploaded to S3.
+	S3 *ETCDSnapshotS3 `json:"s3,omitempty"`
+}
+
+// ETCDSnapshotS3 holds information about the S3 storage system holding the snapshot.
+type ETCDSnapshotS3 struct {
+	// Endpoint is the host or host:port of the S3 service
+	Endpoint string `json:"endpoint,omitempty"`
+	// EndpointCA is the path on disk to the S3 service's trusted CA list. Leave empty to use the OS CA bundle.
+	EndpointCA string `json:"endpointCA,omitempty"`
+	// SkipSSLVerify is true if TLS certificate verification is disabled
+	SkipSSLVerify bool `json:"skipSSLVerify,omitempty"`
+	// Bucket is the bucket holding the snapshot
+	Bucket string `json:"bucket,omitempty"`
+	// Region is the region of the S3 service
+	Region string `json:"region,omitempty"`
+	// Prefix is the prefix in which the snapshot file is stored.
+	Prefix string `json:"prefix,omitempty"`
+	// Insecure is true if the S3 service uses HTTP instead of HTTPS
+	Insecure bool `json:"insecure,omitempty"`
+}
+
+// ETCDSnapshotStatus is the status of the ETCDSnapshotFile object.
+type ETCDSnapshotStatus struct {
+	// Size is the size of the snapshot file, in bytes. If not specified, the snapshot failed.
+	Size *resource.Quantity `json:"size,omitempty" column:""`
+	// CreationTime is the timestamp when the snapshot was taken by etcd.
+	CreationTime *metav1.Time `json:"creationTime,omitempty" column:""`
+	// ReadyToUse indicates that the snapshot is available to be restored.
+	ReadyToUse *bool `json:"readyToUse,omitempty"`
+	// Error is the last observed error during snapshot creation, if any.
+	// If the snapshot is retried, this field will be cleared on success.
+	Error *ETCDSnapshotError `json:"error,omitempty"`
+}
+
+// ETCDSnapshotError describes an error encountered during snapshot creation.
+type ETCDSnapshotError struct {
+	// Time is the timestamp when the error was encountered.
+	Time *metav1.Time `json:"time,omitempty"`
+	// Message is a string detailing the encountered error during snapshot creation if specified.
+	// NOTE: message may be logged, and it should not contain sensitive information.
+	Message *string `json:"message,omitempty"`
 }