Feb 06 09:51:34 production-nomad-consul-server nomad[2505]:     2018/02/06 09:51:34 [WARN] raft: Rejecting vote request from 10.0.21.253:4647 since we have a leader: 10.0.11.177:4647
Feb 06 09:51:36 production-nomad-consul-server nomad[2505]:     2018/02/06 09:51:36.359611 [ERR] worker: failed to dequeue evaluation: rpc error: rpc error: eval broker disabled
Feb 06 16:40:46 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:46 [WARN] raft: Rejecting vote request from 10.0.11.177:4647 since we have a leader: 10.0.21.253:4647
Feb 06 16:40:47 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:47 [WARN] raft: Heartbeat timeout from "10.0.21.253:4647" reached, starting election
Feb 06 16:40:47 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:47 [INFO] raft: Node at 10.0.31.24:4647 [Candidate] entering Candidate state in term 30
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [WARN] raft: Election timeout reached, restarting election
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: Node at 10.0.31.24:4647 [Candidate] entering Candidate state in term 31
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: Election won. Tally: 2
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: Node at 10.0.31.24:4647 [Leader] entering Leader state
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: Added peer 10.0.11.177:4647, starting replication
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: Added peer 10.0.21.253:4647, starting replication
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48.065152 [INFO] nomad: cluster leadership acquired
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647}
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48.089439 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48.089456 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [WARN] raft: Rejecting vote request from 10.0.21.253:4647 since we have a leader: 10.0.31.24:4647
Feb 06 16:40:48 production-nomad-consul-server nomad[2505]:     2018/02/06 16:40:48 [INFO] raft: pipelining replication to peer {Voter 10.0.21.253:4647 10.0.21.253:4647}
Feb 06 16:58:50 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:50 [WARN] raft: Failed to contact 10.0.21.253:4647 in 500.191882ms
Feb 06 16:58:51 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:51 [WARN] raft: Failed to contact 10.0.21.253:4647 in 500.223816ms
Feb 06 16:58:52 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:52 [WARN] raft: Failed to contact 10.0.21.253:4647 in 962.775805ms
Feb 06 16:58:53 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:53 [WARN] raft: Failed to contact 10.0.21.253:4647 in 500.234721ms
Feb 06 16:58:54 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:54 [WARN] raft: Failed to contact 10.0.21.253:4647 in 984.942935ms
Feb 06 16:58:55 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:55 [WARN] raft: Failed to contact 10.0.21.253:4647 in 500.172185ms
Feb 06 16:58:55 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:55 [WARN] raft: Failed to contact 10.0.21.253:4647 in 957.872619ms
Feb 06 16:58:56 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:56 [WARN] raft: Failed to contact 10.0.21.253:4647 in 1.447751009s
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59 [WARN] raft: Rejecting vote request from 10.0.21.253:4647 since we have a leader: 10.0.31.24:4647
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59 [WARN] raft: Rejecting vote request from 10.0.21.253:4647 since we have a leader: 10.0.31.24:4647
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59 [ERR] raft: peer {Voter 10.0.21.253:4647 10.0.21.253:4647} has newer term, stopping replication
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.21.253:4647 10.0.21.253:4647}
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59 [INFO] raft: Node at 10.0.31.24:4647 [Follower] entering Follower state (Leader: "")
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647}
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59.781899 [INFO] nomad: cluster leadership lost
Feb 06 16:58:59 production-nomad-consul-server nomad[2505]:     2018/02/06 16:58:59.781942 [ERR] worker: failed to dequeue evaluation: eval broker disabled
Feb 06 16:59:12 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:12 [WARN] raft: Rejecting vote request from 10.0.11.177:4647 since we have a leader: 10.0.21.253:4647
Feb 06 16:59:12 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:12 [WARN] raft: Heartbeat timeout from "10.0.21.253:4647" reached, starting election
Feb 06 16:59:12 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:12 [INFO] raft: Node at 10.0.31.24:4647 [Candidate] entering Candidate state in term 37
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [WARN] raft: Election timeout reached, restarting election
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: Node at 10.0.31.24:4647 [Candidate] entering Candidate state in term 38
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: Election won. Tally: 2
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: Node at 10.0.31.24:4647 [Leader] entering Leader state
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: Added peer 10.0.11.177:4647, starting replication
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: Added peer 10.0.21.253:4647, starting replication
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13.645832 [INFO] nomad: cluster leadership acquired
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647}
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [WARN] raft: Rejecting vote request from 10.0.21.253:4647 since we have a leader: 10.0.31.24:4647
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [INFO] raft: pipelining replication to peer {Voter 10.0.21.253:4647 10.0.21.253:4647}
Feb 06 16:59:13 production-nomad-consul-server nomad[2505]:     2018/02/06 16:59:13 [WARN] raft: Rejecting vote request from 10.0.21.253:4647 since we have a leader: 10.0.31.24:4647