restore dqlite state from snapshot

[karawitan]

Hi !

I have accidentally removed some CRD on my dev cluster .. looks like dqlite is snapshotting data, how is it possible to recover from a previous state ?

ubuntu@n9:~/ceph/rook/deploy/examples$ date
Wed Sep 27 15:00:54 UTC 2023
ubuntu@n9:~/ceph/rook/deploy/examples$ ls -ltr /var/snap/microk8s/current/var/kubernetes/backend/
total 160808
-rw-rw---- 1 root microk8s    1862 Sep 11 15:15 cluster.crt
-rw-rw---- 1 root microk8s    3272 Sep 11 15:15 cluster.key
-rw-rw---- 1 root microk8s      62 Sep 11 15:15 info.yaml
-rw-rw---- 1 root microk8s      32 Sep 27 14:06 metadata1
-rw-rw---- 1 root microk8s      32 Sep 27 14:06 metadata2
-rw-rw---- 1 root microk8s      68 Sep 27 14:06 localnode.yaml
srw-rw---- 1 root microk8s       0 Sep 27 14:06 kine.sock:12379
-rw-rw-r-- 1 root microk8s       2 Sep 27 14:09 failure-domain
-rw-rw---- 1 root microk8s 8385952 Sep 27 14:47 0000000001313821-0000000001314530
-rw-rw---- 1 root microk8s 3588992 Sep 27 14:47 0000000001314531-0000000001314844
-rw-rw---- 1 root microk8s 8375424 Sep 27 14:48 0000000001314845-0000000001315463
-rw-rw---- 1 root microk8s 8375480 Sep 27 14:50 0000000001315464-0000000001315799
-rw-rw---- 1 root microk8s 5327680 Sep 27 14:50 0000000001315800-0000000001315868
-rw-rw---- 1 root microk8s 8359696 Sep 27 14:51 0000000001315869-0000000001316041
-rw-rw---- 1 root microk8s 8380904 Sep 27 14:51 0000000001316042-0000000001316623
-rw-rw---- 1 root microk8s 3121848 Sep 27 14:51 0000000001316624-0000000001316892
-rw-rw---- 1 root microk8s 8361776 Sep 27 14:52 0000000001316893-0000000001317606
-rw-rw---- 1 root microk8s 3834944 Sep 27 14:52 0000000001317607-0000000001317916
-rw-rw---- 1 root microk8s 8369336 Sep 27 14:52 0000000001317917-0000000001318507
-rw-rw---- 1 root microk8s 8367376 Sep 27 14:54 0000000001318508-0000000001318841
-rw-rw---- 1 root microk8s 2026248 Sep 27 14:55 0000000001318842-0000000001318940
-rw-rw---- 1 root microk8s 8371128 Sep 27 14:56 0000000001318941-0000000001319498
-rw-rw---- 1 root microk8s 5302976 Sep 27 14:56 0000000001319499-0000000001319964
-rw-rw---- 1 root microk8s 8379496 Sep 27 14:57 0000000001319965-0000000001320697
-rw-rw---- 1 root microk8s 3254792 Sep 27 14:57 0000000001320698-0000000001320988
-rw-rw---- 1 root microk8s     128 Sep 27 14:57 snapshot-21-1320988-3156331.meta
-rw-rw---- 1 root microk8s 4596149 Sep 27 14:57 snapshot-21-1320988-3156331
-rw-rw---- 1 root microk8s 8387264 Sep 27 14:57 0000000001320989-0000000001321714
-rw-rw---- 1 root microk8s 3366104 Sep 27 14:57 0000000001321715-0000000001322012
-rw-rw---- 1 root microk8s     128 Sep 27 14:57 snapshot-21-1322012-3170889.meta
-rw-rw---- 1 root microk8s 8388608 Sep 27 14:57 open-84
-rw-rw---- 1 root microk8s 4499218 Sep 27 14:57 snapshot-21-1322012-3170889
-rw-rw---- 1 root microk8s 8388608 Sep 27 14:59 open-85
-rw-rw---- 1 root microk8s 8387112 Sep 27 14:59 0000000001322013-0000000001322507
-rw-rw---- 1 root microk8s     202 Sep 27 15:00 cluster.yaml
-rw-rw---- 1 root microk8s 8388608 Sep 27 15:01 open-83

[cole-miller]

If I understand correctly, you performed a microk8s operation that resulted in some microk8s data that's stored on dqlite being deleted from the database, and you want to recover it. Effectively, you want to roll the dqlite database back to its state just before you deleted the data. Is that about right?

dqlite doesn't have such a rollback function built in, so this is going to be a fairly manual process. The first thing I would do is to try to figure out whether either of the two snapshots that you see in that ls output contain the data that you care about. That would mean starting microk8s with a dqlite data directory that contains just these files from your list:

-rw-rw---- 1 root microk8s    1862 Sep 11 15:15 cluster.crt
-rw-rw---- 1 root microk8s    3272 Sep 11 15:15 cluster.key
-rw-rw---- 1 root microk8s      62 Sep 11 15:15 info.yaml
-rw-rw---- 1 root microk8s      32 Sep 27 14:06 metadata1
-rw-rw---- 1 root microk8s      32 Sep 27 14:06 metadata2
-rw-rw---- 1 root microk8s      68 Sep 27 14:06 localnode.yaml
srw-rw---- 1 root microk8s       0 Sep 27 14:06 kine.sock:12379
-rw-rw-r-- 1 root microk8s       2 Sep 27 14:09 failure-domain
-rw-rw---- 1 root microk8s     128 Sep 27 14:57 snapshot-21-1320988-3156331.meta
-rw-rw---- 1 root microk8s 4596149 Sep 27 14:57 snapshot-21-1320988-3156331
-rw-rw---- 1 root microk8s     202 Sep 27 15:00 cluster.yaml

(or pick the other matched pair of snapshot files). Don't just destroy the other files obviously, make a copy of the data directory first. I don't know whether you're running a HA configuration (multiple nodes), but if so it would be good to see the listing of files on those nodes as well. Otherwise, you should be able to start microk8s normally on the single node with this modified data directory and query it to see whether the CRD is there. If so, you can copy it somehow, then go back to the pristine data directory and re-add it to your running cluster. If not, please report back and I can help you look elsewhere for the missing data.