Force-Restore quorum after 1/2 nodes disappears?

[IngwiePhoenix]

Long story short: I used two nodes to bootstrap my cluster, and one got zapped (faulty microSD plugged in via a USB reader seemed to have knocked the poor SBC stupid and now it no longer boots/functions). Eversince that, I get errors from Raft trying to connect to that dead node and thus k3s no longer boots ... which is a bit of a problem.

How do I force-reset all peers in the cluster but keep the current etcd store (deployments, pvc links, ...) so that I can boot that node again and then apply updates to the deployments to point to the right IPs again? The node that imploded was also my NAS - so the IP for that has changed now and needs to be updated in the nfs-subdir-csi.

Thanks!

[shakibamoshiri]

since in etcd a quorum of an odd number is needed, 1, 3, ,5 etc and yours is 2, I think and guess having 2 does not give you any kind of HA, and better to stick to either 1 or 3

for the issue , you can restore the etcd snapshot that k3s server automatically creates every 6 hours

please notice restring a snapshot is destructive and do it on a test server first

[IngwiePhoenix]

I can see the snapshots fine:

root@cluserboi /v/l/r/k/s/db# tree -a
.
├── etcd
│   ├── config
│   ├── member
│   │   ├── snap
│   │   │   ├── 000000000000050d-0000000004e4e508.snap
│   │   │   ├── 000000000000050d-0000000004e50c19.snap
│   │   │   ├── 000000000000050e-0000000004e5332a.snap
│   │   │   ├── 000000000000050f-0000000004e55a3b.snap
│   │   │   ├── 000000000000050f-0000000004e5814d.snap
│   │   │   └── db
│   │   └── wal
│   │       ├── 0000000000000105-000000000141b3cd.wal.broken
│   │       ├── 000000000000011b-00000000015a051f.wal.broken
│   │       ├── 000000000000011c-00000000015ad136.wal.broken
│   │       ├── 0000000000000120-00000000015d6f11.wal.broken
│   │       ├── 0000000000000129-0000000001668078.wal.broken
│   │       ├── 000000000000019f-0000000001e8a1c9.wal.broken
│   │       ├── 0000000000000999-0000000004e3bb68.wal
│   │       ├── 000000000000099a-0000000004e42875.wal
│   │       ├── 000000000000099b-0000000004e49781.wal
│   │       ├── 000000000000099c-0000000004e50bf5.wal
│   │       ├── 000000000000099d-0000000004e55996.wal
│   │       ├── 0.tmp
│   │       └── 1.tmp
│   └── name
└── snapshots
    ├── etcd-snapshot-clusterboi-1724539134.zip
    ├── etcd-snapshot-clusterboi-1724580000.zip
    ├── etcd-snapshot-clusterboi-1724623203.zip
    ├── etcd-snapshot-clusterboi-1724666402.zip
    └── etcd-snapshot-clusterboi-1724711790.zip

6 directories, 26 files

Do I just unpack the contents of any of the zips straight into etcd/member? Looking at the contents of the most recent one gives me a file whose name does not seem to match any of the files within etc/member...

root@cluserboi /v/l/r/k/s/db# unzip -l snapshots/etcd-snapshot-clusterboi-1724711790.zip
Archive:  snapshots/etcd-snapshot-clusterboi-1724711790.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
 70078496  2024-08-27 00:36   etcd-snapshot-clusterboi-1724711790
---------                     -------
 70078496                     1 file

As for the destructiveness; that's fine, as long as I can get my cluster back ... or, well, my node. It'll take some time untill I can get a replacement, but I mainly run k3s to improve my knowledge of Kubernetes for work (where our cluster is at 3 and I understand now why. xD)

[shakibamoshiri]

they are saved at /var/lib/rancher/k3s/server/db/snapshots/ for me version 1.30.3 is not zip formatted . Seems for you are different. Yes this is the etcd-snapshot-clusterboi-1724711790 snapshot you should restore

[IngwiePhoenix]

I initially started with 1.28, so that might be why it's a zip. Alright, will give it a shot and see!

[brandond]

You don't need to do a restore. Just start one of the servers with --cluster-reset, and it will reset the etcd cluster membership back to only that node. At that point you can follow the instructions to remove the DB from the other nodes and join additional etcd cluster members. Just make sure you have an odd number this time.

[IngwiePhoenix]

I managed to do it with an etcd snapshot restore. Was one heck of a troubble, but it worked out. The values from the config and name file within the "old" etcd folder helped a lot.

etcdctl snapshot restore \
  --cacert=/var/lib/rancher/k3s/server/tls/etcd/server-ca.crt \
  --cert=/var/lib/rancher/k3s/server/tls/etcd/server-client.crt \
  --data-dir=/var/lib/rancher/k3s/server/db/etcd \
  --wal-dir=/var/lib/rancher/k3s/server/db/etcd/member/wal \
  /root/k3s-recover/etcd-snapshot-clusterboi-1724711790 \
  --initial-cluster="clusterboi-e6462b43=https://192.168.1.3:2380" \
  --name=clusterboi-e6462b43 \
  --initial-advertise-peer-urls=https://192.168.1.3:2380

Not the cleanest of works - but one that most certainly worked. The error messages provided by etcdctl helped me find the last two parameters about the initial cluster.
Thanks for the help!

[IngwiePhoenix]

I completely overlooked that...

root@cluserboi ~# k3s server --help | grep -i reset
   --cluster-reset                            (cluster) Forget all peers and become sole member of a new cluster [$K3S_CLUSTER_RESET]
   --cluster-reset-restore-path value         (db) Path to snapshot file to be restored

Welp, now I know, I guess x)

[brandond]

I would also recommend against using etcdctl to restore to directly restore a snapshot into the managed etcd datastore. This is not intended to be something you do by hand, use the cluster-reset and cluster-reset-restore-path to do that.