etcd cluster losing consensus

[jhgg]

We run a 3 node etcd cluster, running since Jan 2016 just fine, with out much changes (aside from updating to etcd 2.3.0). Around the time of the failure, we see these log-lines on the servers:

etcd-prd-1-1

016-08-26 22:23:14.403390 I | fileutil: purged file /usr/local/var/run/etcd/member/snap/0000000000000002-00000000009a211b.snap successfully
2016-08-26 22:30:46.100648 I | raft: 8d7c6a52ff1dd17e is starting a new election at term 2
2016-08-26 22:30:46.100804 I | raft: 8d7c6a52ff1dd17e became candidate at term 3
2016-08-26 22:30:46.100842 I | raft: 8d7c6a52ff1dd17e received vote from 8d7c6a52ff1dd17e at term 3
2016-08-26 22:30:46.100886 I | raft: 8d7c6a52ff1dd17e [logterm: 2, index: 10157693] sent vote request to 4810d4356fa22a49 at term 3
2016-08-26 22:30:46.100906 I | raft: 8d7c6a52ff1dd17e [logterm: 2, index: 10157693] sent vote request to eb9c9807f4feeb26 at term 3
2016-08-26 22:30:46.100947 I | raft: raft.node: 8d7c6a52ff1dd17e lost leader eb9c9807f4feeb26 at term 3
2016-08-26 22:30:46.106525 I | raft: 8d7c6a52ff1dd17e received vote from 4810d4356fa22a49 at term 3
2016-08-26 22:30:46.106570 I | raft: 8d7c6a52ff1dd17e [quorum:2] has received 2 votes and 0 vote rejections
2016-08-26 22:30:46.106626 I | raft: 8d7c6a52ff1dd17e became leader at term 3
2016-08-26 22:30:46.106654 I | raft: raft.node: 8d7c6a52ff1dd17e elected leader 8d7c6a52ff1dd17e at term 3
2016-08-26 22:30:47.189135 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.189280 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.314452 W | etcdserver: failed to reach the peerURL(http://etcd-prd-1-3:2380) of member eb9c9807f4feeb26 (Get http://etcd-prd-1-3:2380/version: dial tcp 10.10.0.17:2380: i/o timeout)
2016-08-26 22:30:47.314546 W | etcdserver: cannot get the version of member eb9c9807f4feeb26 (Get http://etcd-prd-1-3:2380/version: dial tcp 10.10.0.17:2380: i/o timeout)
2016-08-26 22:30:47.474298 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.474435 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.474494 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.475969 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.476052 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.575732 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.585210 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.593943 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.604141 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.675729 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.775882 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.843421 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.876025 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.876163 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.975807 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.075637 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.175905 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.275743 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.302904 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.320538 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.320621 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.324407 I | raft: 8d7c6a52ff1dd17e [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:49.878816 I | raft: 8d7c6a52ff1dd17e [term: 3] received a MsgVote message with higher term from eb9c9807f4feeb26 [term: 4]
2016-08-26 22:30:49.878931 I | raft: 8d7c6a52ff1dd17e became follower at term 4
2016-08-26 22:30:49.878987 I | raft: 8d7c6a52ff1dd17e [logterm: 3, index: 10157723, vote: 0] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 4
2016-08-26 22:30:49.879014 I | raft: raft.node: 8d7c6a52ff1dd17e lost leader 8d7c6a52ff1dd17e at term 4
2016-08-26 22:30:50.978460 I | raft: 8d7c6a52ff1dd17e [term: 4] received a MsgVote message with higher term from eb9c9807f4feeb26 [term: 5]
2016-08-26 22:30:50.978616 I | raft: 8d7c6a52ff1dd17e became follower at term 5
2016-08-26 22:30:50.978648 I | raft: 8d7c6a52ff1dd17e [logterm: 3, index: 10157723, vote: 0] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 5
2016-08-26 22:30:51.984444 I | raft: 8d7c6a52ff1dd17e [term: 5] received a MsgVote message with higher term from 4810d4356fa22a49 [term: 6]
2016-08-26 22:30:51.984573 I | raft: 8d7c6a52ff1dd17e became follower at term 6
2016-08-26 22:30:51.984633 I | raft: 8d7c6a52ff1dd17e [logterm: 3, index: 10157723, vote: 0] voted for 4810d4356fa22a49 [logterm: 3, index: 10157723] at term 6
2016-08-26 22:30:51.987905 I | raft: raft.node: 8d7c6a52ff1dd17e elected leader 4810d4356fa22a49 at term 6
2016-08-26 22:30:52.078471 I | raft: 8d7c6a52ff1dd17e [logterm: 6, index: 10157745, vote: 4810d4356fa22a49] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 6
2016-08-26 22:30:52.104081 E | etcdhttp: got unexpected response error (etcdserver: request timed out)
2016-08-26 22:30:53.278923 I | raft: 8d7c6a52ff1dd17e [term: 6] received a MsgVote message with higher term from eb9c9807f4feeb26 [term: 7]
2016-08-26 22:30:53.279093 I | raft: 8d7c6a52ff1dd17e became follower at term 7
2016-08-26 22:30:53.279155 I | raft: 8d7c6a52ff1dd17e [logterm: 6, index: 10157756, vote: 0] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 7
2016-08-26 22:30:53.279194 I | raft: raft.node: 8d7c6a52ff1dd17e lost leader 4810d4356fa22a49 at term 7
2016-08-26 22:30:53.899182 E | etcdhttp: got unexpected response error (etcdserver: request timed out) [merged 6 repeated lines in 1.8s]
2016-08-26 22:30:54.400683 I | raft: 8d7c6a52ff1dd17e is starting a new election at term 7
2016-08-26 22:30:54.400803 I | raft: 8d7c6a52ff1dd17e became candidate at term 8
2016-08-26 22:30:54.400837 I | raft: 8d7c6a52ff1dd17e received vote from 8d7c6a52ff1dd17e at term 8

etcd-prd-1-2

016-08-26 22:30:46.103776 I | raft: 4810d4356fa22a49 [term: 2] received a MsgVote message with higher term from 8d7c6a52ff1dd17e [term: 3]
2016-08-26 22:30:46.103957 I | raft: 4810d4356fa22a49 became follower at term 3
2016-08-26 22:30:46.104005 I | raft: 4810d4356fa22a49 [logterm: 2, index: 10157693, vote: 0] voted for 8d7c6a52ff1dd17e [logterm: 2, index: 10157693] at term 3
2016-08-26 22:30:46.104033 I | raft: raft.node: 4810d4356fa22a49 lost leader eb9c9807f4feeb26 at term 3
2016-08-26 22:30:46.107146 I | raft: raft.node: 4810d4356fa22a49 elected leader 8d7c6a52ff1dd17e at term 3
2016-08-26 22:30:47.189104 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.189251 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.474361 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.474460 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.474546 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.475978 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.476059 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.575754 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.585208 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.594328 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.604090 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.676037 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.775811 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.843471 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.875927 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.876064 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:47.975804 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.075736 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.175852 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.275689 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgHeartbeat message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.302913 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.320634 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.320733 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:48.324425 I | raft: 4810d4356fa22a49 [term: 3] ignored a MsgApp message with lower term from eb9c9807f4feeb26 [term: 2]
2016-08-26 22:30:49.878649 I | raft: 4810d4356fa22a49 [term: 3] received a MsgVote message with higher term from eb9c9807f4feeb26 [term: 4]
2016-08-26 22:30:49.878739 I | raft: 4810d4356fa22a49 became follower at term 4
2016-08-26 22:30:49.878802 I | raft: 4810d4356fa22a49 [logterm: 3, index: 10157723, vote: 0] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 4
2016-08-26 22:30:49.878844 I | raft: raft.node: 4810d4356fa22a49 lost leader 8d7c6a52ff1dd17e at term 4
2016-08-26 22:30:50.978349 I | raft: 4810d4356fa22a49 [term: 4] received a MsgVote message with higher term from eb9c9807f4feeb26 [term: 5]
2016-08-26 22:30:50.978527 I | raft: 4810d4356fa22a49 became follower at term 5
2016-08-26 22:30:50.978587 I | raft: 4810d4356fa22a49 [logterm: 3, index: 10157723, vote: 0] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 5
2016-08-26 22:30:51.981532 I | raft: 4810d4356fa22a49 is starting a new election at term 5
2016-08-26 22:30:51.981635 I | raft: 4810d4356fa22a49 became candidate at term 6
2016-08-26 22:30:51.981677 I | raft: 4810d4356fa22a49 received vote from 4810d4356fa22a49 at term 6
2016-08-26 22:30:51.981725 I | raft: 4810d4356fa22a49 [logterm: 3, index: 10157723] sent vote request to 8d7c6a52ff1dd17e at term 6
2016-08-26 22:30:51.981746 I | raft: 4810d4356fa22a49 [logterm: 3, index: 10157723] sent vote request to eb9c9807f4feeb26 at term 6
2016-08-26 22:30:51.987070 I | raft: 4810d4356fa22a49 received vote from 8d7c6a52ff1dd17e at term 6
2016-08-26 22:30:51.987151 I | raft: 4810d4356fa22a49 [quorum:2] has received 2 votes and 0 vote rejections
2016-08-26 22:30:51.987222 I | raft: 4810d4356fa22a49 became leader at term 6
2016-08-26 22:30:51.987307 I | raft: raft.node: 4810d4356fa22a49 elected leader 4810d4356fa22a49 at term 6
2016-08-26 22:30:52.078389 I | raft: 4810d4356fa22a49 [logterm: 6, index: 10157745, vote: 4810d4356fa22a49] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 6
2016-08-26 22:30:52.302789 E | etcdhttp: got unexpected response error (etcdserver: request timed out)
2016-08-26 22:30:53.278875 I | raft: 4810d4356fa22a49 [term: 6] received a MsgVote message with higher term from eb9c9807f4feeb26 [term: 7]
2016-08-26 22:30:53.278995 I | raft: 4810d4356fa22a49 became follower at term 7
2016-08-26 22:30:53.279058 I | raft: 4810d4356fa22a49 [logterm: 6, index: 10157756, vote: 0] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 7
2016-08-26 22:30:53.279084 I | raft: raft.node: 4810d4356fa22a49 lost leader 4810d4356fa22a49 at term 7
2016-08-26 22:30:53.872098 E | etcdhttp: got unexpected response error (etcdserver: request timed out) [merged 5 repeated lines in 1.57s]
2016-08-26 22:30:54.403682 I | raft: 4810d4356fa22a49 [term: 7] received a MsgVote message with higher term from 8d7c6a52ff1dd17e [term: 8]
2016-08-26 22:30:54.403802 I | raft: 4810d4356fa22a49 became follower at term 8
2016-08-26 22:30:54.403858 I | raft: 4810d4356fa22a49 [logterm: 6, index: 10157756, vote: 0] voted for 8d7c6a52ff1dd17e [logterm: 6, index: 10157756] at term 8
2016-08-26 22:30:54.407115 I | raft: raft.node: 4810d4356fa22a49 elected leader 8d7c6a52ff1dd17e at term 8
2016-08-26 22:30:54.678107 I | raft: 4810d4356fa22a49 [logterm: 8, index: 10157776, vote: 8d7c6a52ff1dd17e] rejected vote from eb9c9807f4feeb26 [logterm: 2, index: 10157706] at term 8

etcd-prd-1-3

016-08-26 22:30:47.181660 W | etcdserver: failed to send out heartbeat on time (deadline exceeded for 2.206153235s)
2016-08-26 22:30:47.181941 W | etcdserver: server is likely overloaded
2016-08-26 22:30:47.182036 W | etcdserver: failed to send out heartbeat on time (deadline exceeded for 2.206559003s)
2016-08-26 22:30:47.182053 W | etcdserver: server is likely overloaded
2016-08-26 22:30:47.190095 W | etcdserver: failed to reach the peerURL(http://etcd-prd-1-2:2380) of member 4810d4356fa22a49 (Get http://etcd-prd-1-2:2380/version: read tcp 10.10.0.17:37836->10.10.0.16:2380: read: connection reset by peer)
2016-08-26 22:30:47.190190 W | etcdserver: cannot get the version of member 4810d4356fa22a49 (Get http://etcd-prd-1-2:2380/version: read tcp 10.10.0.17:37836->10.10.0.16:2380: read: connection reset by peer)
2016-08-26 22:30:47.191280 W | etcdserver: failed to reach the peerURL(http://etcd-prd-1-1:2380) of member 8d7c6a52ff1dd17e (Get http://etcd-prd-1-1:2380/version: read tcp 10.10.0.17:46260->10.10.0.13:2380: read: connection reset by peer)
2016-08-26 22:30:47.191339 W | etcdserver: cannot get the version of member 8d7c6a52ff1dd17e (Get http://etcd-prd-1-1:2380/version: read tcp 10.10.0.17:46260->10.10.0.13:2380: read: connection reset by peer)
2016-08-26 22:30:47.473523 W | etcdserver: failed to send out heartbeat on time (deadline exceeded for 91.888953ms)
2016-08-26 22:30:47.473602 W | etcdserver: server is likely overloaded
2016-08-26 22:30:47.473637 W | etcdserver: failed to send out heartbeat on time (deadline exceeded for 91.601626ms)
2016-08-26 22:30:47.473651 W | etcdserver: server is likely overloaded
2016-08-26 22:30:48.320745 I | raft: eb9c9807f4feeb26 [term: 2] received a MsgVote message with higher term from 8d7c6a52ff1dd17e [term: 3]
2016-08-26 22:30:48.321017 I | raft: eb9c9807f4feeb26 became follower at term 3
2016-08-26 22:30:48.322088 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706, vote: 0] rejected vote from 8d7c6a52ff1dd17e [logterm: 2, index: 10157693] at term 3
2016-08-26 22:30:48.322150 I | raft: raft.node: eb9c9807f4feeb26 lost leader eb9c9807f4feeb26 at term 3
2016-08-26 22:30:49.875440 I | raft: eb9c9807f4feeb26 is starting a new election at term 3
2016-08-26 22:30:49.875573 I | raft: eb9c9807f4feeb26 became candidate at term 4
2016-08-26 22:30:49.875652 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 4
2016-08-26 22:30:49.875714 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 4
2016-08-26 22:30:49.875737 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 4
2016-08-26 22:30:50.975488 I | raft: eb9c9807f4feeb26 is starting a new election at term 4
2016-08-26 22:30:50.975633 I | raft: eb9c9807f4feeb26 became candidate at term 5
2016-08-26 22:30:50.975684 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 5
2016-08-26 22:30:50.975709 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 5
2016-08-26 22:30:50.975727 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 5
2016-08-26 22:30:52.075478 I | raft: eb9c9807f4feeb26 is starting a new election at term 5
2016-08-26 22:30:52.075625 I | raft: eb9c9807f4feeb26 became candidate at term 6
2016-08-26 22:30:52.075676 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 6
2016-08-26 22:30:52.075700 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 6
2016-08-26 22:30:52.075770 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 6
2016-08-26 22:30:53.275475 I | raft: eb9c9807f4feeb26 is starting a new election at term 6
2016-08-26 22:30:53.275631 I | raft: eb9c9807f4feeb26 became candidate at term 7
2016-08-26 22:30:53.275683 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 7
2016-08-26 22:30:53.275718 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 7
2016-08-26 22:30:53.275792 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 7
2016-08-26 22:30:54.225269 E | etcdhttp: got unexpected response error (etcdserver: request timed out)
2016-08-26 22:30:54.675559 I | raft: eb9c9807f4feeb26 is starting a new election at term 7
2016-08-26 22:30:54.675658 I | raft: eb9c9807f4feeb26 became candidate at term 8
2016-08-26 22:30:54.675690 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 8
2016-08-26 22:30:54.675761 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 8
2016-08-26 22:30:54.675809 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 8
2016-08-26 22:30:55.868679 E | etcdhttp: got unexpected response error (etcdserver: request timed out) [merged 12 repeated lines in 1.64s]
2016-08-26 22:30:55.975433 I | raft: eb9c9807f4feeb26 is starting a new election at term 8
2016-08-26 22:30:55.975543 I | raft: eb9c9807f4feeb26 became candidate at term 9
2016-08-26 22:30:55.975623 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 9
2016-08-26 22:30:55.975660 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 9
2016-08-26 22:30:55.975722 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 9
2016-08-26 22:30:56.868689 E | etcdhttp: got unexpected response error (etcdserver: request timed out) [merged 4 repeated lines in 1s]
2016-08-26 22:30:57.375473 I | raft: eb9c9807f4feeb26 is starting a new election at term 9
2016-08-26 22:30:57.375618 I | raft: eb9c9807f4feeb26 became candidate at term 10
2016-08-26 22:30:57.375653 I | raft: eb9c9807f4feeb26 received vote from eb9c9807f4feeb26 at term 10
2016-08-26 22:30:57.375690 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 4810d4356fa22a49 at term 10
2016-08-26 22:30:57.375713 I | raft: eb9c9807f4feeb26 [logterm: 2, index: 10157706] sent vote request to 8d7c6a52ff1dd17e at term 10
2016-08-26 22:30:58.475473 I | raft: eb9c9807f4feeb26 is starting a new election at term 10
2016-08-26 22:30:58.475587 I | raft: eb9c9807f4feeb26 became candidate at term 11

Can anyone provide any pointers as to where we can look? Cluster has been completely fine for months, but this kinda thing has happened twice in the past 2 weeks. CPU usage/load on the nodes were normal according to our metrics during this time.

[xiang90]

@jhgg What do you mean by losing consensus? Do you observe inconsistent data? Or you mean there are some leader elections?

[jhgg]

The data seems to be consistent, it just stops accepting writes as it cannot elect a leader anymore. It basically gets stuck voting.

[xiang90]

@jhgg How did you solve the stuck issue? By restarting the cluster? Or it recovered itself?

[jhgg]

We have a script that basically reads from one of the reachable nodes in the cluster, retrieving the the contents of all the keys recursively and dumping them to a json file, firewalling the cluster off from the rest of the network (our services compensate by using stale data for discovery when the cluster is un-reachable) - deleting the data directories and starting a new cluster, then writing the old discovery data back to the cluster before making them reachable again.

[jhgg]

Script here: https://gist.github.com/jhgg/6a567c725640785752e24bc9007fa1aa

[xiang90]

@jhgg The problem seems to be identical with #4888. And it is solved in 2.3.2 release.

[jhgg]

Interesting. We will perform an upgrade to the latest 2.3.x release tonight and see if it happens anymore.

[xiang90]

@jhgg I close this issue. Please reopen if it happens again after the upgrades. You can directly upgrades from 2.3.0 -> 2.3.7. Thanks for reporting.

[jhgg]

Thanks for the quick response @xiang90 ❤️