clientv3: snapshot status does not check database integrity

[jingyih]

Version
etcd Version: 3.3.0+git
Git SHA: f32bc50
Go Version: go1.10.3
Go OS/Arch: linux/amd64

Issue
We have a corrupted snapshot file, identifiable by running bbolt check tool. When users do etcdctl snapshot status, it should notify user if the snapshot file is corrupted.
I will send out a PR to add database integrity verification in snapshot status.

Example output

etcdctl --write-out=table snapshot status corrupt_db/backup.db
+----------+----------+------------+------------+
|   HASH   | REVISION | TOTAL KEYS | TOTAL SIZE |
+----------+----------+------------+------------+
| 1d0551d2 | 45565687 |        709 |     8.4 MB |
+----------+----------+------------+------------+

bbolt check corrupt_db/backup.db 
page 961: already freed
page 962: already freed
page 1347: already freed
page 1442: already freed
page 1760: already freed
page 2023: already freed
page 2033: already freed
page 969: unreachable unfreed
page 970: unreachable unfreed
page 1355: unreachable unfreed
page 1450: unreachable unfreed
page 1768: unreachable unfreed
page 2031: unreachable unfreed
page 2041: unreachable unfreed
14 errors found
invalid value

[jingyih]

/cc @jpbetz @gyuho
Please let me know if you have any comments.

[jpbetz]

I’m strongly in favor of adding the bbolt check to snapshot status.

[xiang90]

@jpbetz @jingyih

A side question for you: is the snapshot file generated by etcd? Have you figured out the root cause of the corruption?

[hexfusion]

@jingyih would it be possible to share this snapshot for the sake of experimentation? Is the underlying data in any way sensitive?

[jpbetz]

@hexfusion we can’t share the one we’ve got on hand, but I can try to produce one that can be shared publicly.

[hexfusion]

@jpbetz, thanks I would appreciate that.

[jingyih]

@xiang90 @hexfusion The corrupted snapshot file is due to freelist corruption. It was fixed and backported all the way to v3.1. The corrupted snapshot file was found on one of our GKE cluster where it is using v3.0.