KV index is smaller than snapshot index

[armstrongli]

I got one case that ETCD couldn't start up caused by error etcdmain: database file (/var/etcd/data/member/snap/db index 11895551) does not match with snapshot (index 13328084).

How did this happen that the KV's index is smaller than the snapshot one?

The full restart logs :

2017-06-14 14:28:53.732637 W | pkg/flags: unrecognized environment variable ETCD_TEST_PORT_2380_TCP_ADDR=192.168.65.11
2017-06-14 14:28:53.732643 W | pkg/flags: unrecognized environment variable ETCD_TEST_PORT_2380_TCP_PORT=2380
2017-06-14 14:28:53.732648 W | pkg/flags: unrecognized environment variable ETCD_TEST_SERVICE_HOST=192.168.65.11
2017-06-14 14:28:53.732653 W | pkg/flags: unrecognized environment variable ETCD_TEST_PORT_4001_TCP=tcp://192.168.65.11:4001
2017-06-14 14:28:53.732658 W | pkg/flags: unrecognized environment variable ETCD_TEST_PORT_2380_TCP_PROTO=tcp
2017-06-14 14:28:53.732663 W | pkg/flags: unrecognized environment variable ETCD_TEST_SERVICE_PORT_SERVERPORT=2380
2017-06-14 14:28:53.732685 I | etcdmain: etcd Version: 3.1.8
2017-06-14 14:28:53.732691 I | etcdmain: Git SHA: d267ca9
2017-06-14 14:28:53.732696 I | etcdmain: Go Version: go1.7.5
2017-06-14 14:28:53.732701 I | etcdmain: Go OS/Arch: linux/amd64
2017-06-14 14:28:53.732707 I | etcdmain: setting maximum number of CPUs to 4, total number of available CPUs is 4
2017-06-14 14:28:53.732753 N | etcdmain: the server is already initialized as member before, starting as etcd member...
2017-06-14 14:28:53.732825 I | embed: listening for peers on http://0.0.0.0:2380
2017-06-14 14:28:53.732863 I | embed: listening for client requests on 0.0.0.0:4001
2017-06-14 14:28:54.733294 W | etcdserver: another etcd process is running with the same data dir and holding the file lock.
2017-06-14 14:28:54.733355 W | etcdserver: waiting for it to exit before starting...
2017-06-14 14:29:10.241685 W | snap: skipped unexpected non snapshot file 0000000000baea71.snap.db
2017-06-14 14:29:10.241723 W | snap: skipped unexpected non snapshot file 0000000000bbd27c.snap.db
2017-06-14 14:29:10.241736 W | snap: skipped unexpected non snapshot file db.tmp
2017-06-14 14:29:10.241746 W | snap: skipped unexpected non snapshot file 0000000000cb5ed4.snap.db
2017-06-14 14:29:10.241751 W | snap: skipped unexpected non snapshot file 0000000000acbe10.snap.db
2017-06-14 14:29:10.258091 I | etcdserver: recovered store from snapshot at index 13328084
2017-06-14 14:29:10.258112 I | etcdserver: name = tess-node-873t0-8911
2017-06-14 14:29:10.258118 I | etcdserver: data dir = /var/etcd/data
2017-06-14 14:29:10.258125 I | etcdserver: member dir = /var/etcd/data/member
2017-06-14 14:29:10.258130 I | etcdserver: heartbeat = 100ms
2017-06-14 14:29:10.258135 I | etcdserver: election = 1000ms
2017-06-14 14:29:10.258140 I | etcdserver: snapshot count = 10000
2017-06-14 14:29:10.258156 I | etcdserver: advertise client URLs = https://tess-node-873t0-8911.51.tess.io:4001
2017-06-14 14:29:16.538855 I | etcdserver: restarting member c2cb0e006c421dd3 in cluster 479149e25b784b97 at commit index 14118834
2017-06-14 14:29:16.590690 I | raft: c2cb0e006c421dd3 became follower at term 1218
2017-06-14 14:29:16.590773 I | raft: newRaft c2cb0e006c421dd3 [peers: [5902a07919e43cdf,b620b4c395187fad,c2cb0e006c421dd3,d49c2ef31cf07365,df3a73af6b1c2179], term: 1218, commit: 14118834, applied: 13328084, lastindex: 14118834, lastterm: 1215]
2017-06-14 14:29:16.591047 I | etcdserver/api: enabled capabilities for version 3.0
2017-06-14 14:29:16.591073 I | etcdserver/membership: added member 5902a07919e43cdf [http://10.65.80.98:2380] to cluster 479149e25b784b97 from store
2017-06-14 14:29:16.591081 I | etcdserver/membership: added member b620b4c395187fad [http://10.65.80.90:2380] to cluster 479149e25b784b97 from store
2017-06-14 14:29:16.591089 I | etcdserver/membership: added member c2cb0e006c421dd3 [http://10.65.80.94:2380] to cluster 479149e25b784b97 from store
2017-06-14 14:29:16.591116 I | etcdserver/membership: added member d49c2ef31cf07365 [http://10.65.80.92:2380] to cluster 479149e25b784b97 from store
2017-06-14 14:29:16.591126 I | etcdserver/membership: added member df3a73af6b1c2179 [http://10.65.80.85:2380] to cluster 479149e25b784b97 from store
2017-06-14 14:29:16.591134 I | etcdserver/membership: set the cluster version to 3.0 from store
2017-06-14 14:29:33.499719 C | etcdmain: database file (/var/etcd/data/member/snap/db index 11895551) does not match with snapshot (index 13328084).

[xiang90]

this should already be fixed by #7834.

searching the previous issue before creating a new one will save you time.