Feb 06 09:51:32 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:32 [WARN] raft: Heartbeat timeout from "10.0.11.177:4647" reached, starting election Feb 06 09:51:32 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:32 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 26 Feb 06 09:51:33 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:33.256223 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled Feb 06 09:51:33 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:33 [ERR] raft-net: Failed to flush response: write tcp 10.0.21.253:4647->10.0.11.177:39450: write: connection reset by peer Feb 06 09:51:33 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:33 [WARN] raft: Election timeout reached, restarting election Feb 06 09:51:33 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:33 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 27 Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: Election won. Tally: 2 Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: Node at 10.0.21.253:4647 [Leader] entering Leader state Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: Added peer 10.0.31.24:4647, starting replication Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: Added peer 10.0.11.177:4647, starting replication Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34.094910 [INFO] nomad: cluster leadership acquired Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: pipelining replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [WARN] raft: Failed to contact 10.0.31.24:4647 in 501.018938ms Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [WARN] raft: Failed to contact 10.0.11.177:4647 in 501.200884ms Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [WARN] raft: Failed to contact quorum of nodes, stepping down Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "") Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34.621292 [INFO] nomad: cluster leadership lost Feb 06 09:51:34 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:34.621341 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [WARN] raft: Heartbeat timeout from "" reached, starting election Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 28 Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: Election won. Tally: 2 Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: Node at 10.0.21.253:4647 [Leader] entering Leader state Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: Added peer 10.0.31.24:4647, starting replication Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: Added peer 10.0.11.177:4647, starting replication Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36.353515 [INFO] nomad: cluster leadership acquired Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: pipelining replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 09:51:36 production-nomad-consul-server nomad[2708]: 2018/02/06 09:51:36 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [WARN] raft: Failed to contact 10.0.31.24:4647 in 500.223257ms Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [WARN] raft: Failed to contact 10.0.31.24:4647 in 510.83952ms Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [WARN] raft: Failed to contact 10.0.11.177:4647 in 500.246812ms Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [WARN] raft: Failed to contact quorum of nodes, stepping down Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "") Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43.227520 [INFO] nomad: cluster leadership lost Feb 06 16:40:43 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:43.227567 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [WARN] raft: Heartbeat timeout from "" reached, starting election Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 29 Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: Election won. Tally: 2 Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: Node at 10.0.21.253:4647 [Leader] entering Leader state Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: Added peer 10.0.31.24:4647, starting replication Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: Added peer 10.0.11.177:4647, starting replication Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44.955274 [INFO] nomad: cluster leadership acquired Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: pipelining replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:40:44 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:44 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [WARN] raft: Failed to contact 10.0.11.177:4647 in 500.235871ms Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [WARN] raft: Failed to contact 10.0.31.24:4647 in 508.570172ms Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [WARN] raft: Failed to contact 10.0.11.177:4647 in 510.51285ms Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [WARN] raft: Failed to contact quorum of nodes, stepping down Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "") Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45.500809 [INFO] nomad: cluster leadership lost Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45.500995 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 16:40:45 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:45.501023 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 16:40:47 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:47 [WARN] raft: Heartbeat timeout from "" reached, starting election Feb 06 16:40:47 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:47 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 30 Feb 06 16:40:48 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:48 [INFO] raft: Duplicate RequestVote for same term: 30 Feb 06 16:40:48 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:48 [INFO] raft: Duplicate RequestVote for same term: 30 Feb 06 16:40:48 production-nomad-consul-server nomad[2708]: 2018/02/06 16:40:48 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "10.0.31.24:4647") Feb 06 16:58:56 production-nomad-consul-server nomad[2708]: 2018/02/06 16:58:56 [WARN] raft: Heartbeat timeout from "10.0.31.24:4647" reached, starting election Feb 06 16:58:56 production-nomad-consul-server nomad[2708]: 2018/02/06 16:58:56 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 32 Feb 06 16:58:58 production-nomad-consul-server nomad[2708]: 2018/02/06 16:58:58 [WARN] raft: Election timeout reached, restarting election Feb 06 16:58:58 production-nomad-consul-server nomad[2708]: 2018/02/06 16:58:58 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 33 Feb 06 16:59:00 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:00 [WARN] raft: Election timeout reached, restarting election Feb 06 16:59:00 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:00 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 34 Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01 [INFO] raft: Election won. Tally: 2 Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01 [INFO] raft: Node at 10.0.21.253:4647 [Leader] entering Leader state Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01 [INFO] raft: Added peer 10.0.31.24:4647, starting replication Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01 [INFO] raft: Added peer 10.0.11.177:4647, starting replication Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01.002395 [INFO] nomad: cluster leadership acquired Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01 [INFO] raft: pipelining replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01.067952 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 16:59:01 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:01 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [WARN] raft: Failed to contact 10.0.31.24:4647 in 500.205978ms Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [WARN] raft: Failed to contact 10.0.31.24:4647 in 510.426914ms Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [WARN] raft: Failed to contact 10.0.11.177:4647 in 509.016674ms Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [WARN] raft: Failed to contact quorum of nodes, stepping down Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "") Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02.458580 [INFO] nomad: cluster leadership lost Feb 06 16:59:02 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:02.458621 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 16:59:03 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:03 [WARN] raft: Heartbeat timeout from "" reached, starting election Feb 06 16:59:03 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:03 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 35 Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [INFO] raft: Election won. Tally: 2 Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [INFO] raft: Node at 10.0.21.253:4647 [Leader] entering Leader state Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [INFO] raft: Added peer 10.0.31.24:4647, starting replication Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [INFO] raft: Added peer 10.0.11.177:4647, starting replication Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04.032387 [INFO] nomad: cluster leadership acquired Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [INFO] raft: pipelining replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:59:04 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:04 [WARN] raft: Failed to contact 10.0.31.24:4647 in 500.238929ms Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [WARN] raft: Failed to contact 10.0.31.24:4647 in 500.123019ms Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [WARN] raft: Failed to contact 10.0.31.24:4647 in 510.389097ms Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [WARN] raft: Failed to contact 10.0.11.177:4647 in 510.138787ms Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [WARN] raft: Failed to contact quorum of nodes, stepping down Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "") Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07.807317 [INFO] nomad: cluster leadership lost Feb 06 16:59:07 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:07.807394 [ERR] worker: failed to dequeue evaluation: eval broker disabled Feb 06 16:59:08 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:08 [WARN] raft: Heartbeat timeout from "" reached, starting election Feb 06 16:59:08 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:08 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 36 Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: Election won. Tally: 2 Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: Node at 10.0.21.253:4647 [Leader] entering Leader state Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: Added peer 10.0.31.24:4647, starting replication Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: Added peer 10.0.11.177:4647, starting replication Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10.141195 [INFO] nomad: cluster leadership acquired Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: pipelining replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: pipelining replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [WARN] raft: Failed to contact 10.0.11.177:4647 in 500.204409ms Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [WARN] raft: Failed to contact 10.0.31.24:4647 in 510.22641ms Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [WARN] raft: Failed to contact 10.0.11.177:4647 in 510.440588ms Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [WARN] raft: Failed to contact quorum of nodes, stepping down Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "") Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10.665287 [ERR] nomad: failed to establish leadership: unable to reconcile job summaries: reconciliation of job summaries failed: leadership lost while committing log Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10.665298 [INFO] nomad: cluster leadership lost Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.11.177:4647 10.0.11.177:4647} Feb 06 16:59:10 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:10 [INFO] raft: aborting pipeline replication to peer {Voter 10.0.31.24:4647 10.0.31.24:4647} Feb 06 16:59:11 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:11 [WARN] raft: Heartbeat timeout from "" reached, starting election Feb 06 16:59:11 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:11 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 37 Feb 06 16:59:13 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:13 [WARN] raft: Election timeout reached, restarting election Feb 06 16:59:13 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:13 [INFO] raft: Node at 10.0.21.253:4647 [Candidate] entering Candidate state in term 38 Feb 06 16:59:13 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:13 [INFO] raft: Duplicate RequestVote for same term: 38 Feb 06 16:59:13 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:13 [INFO] raft: Node at 10.0.21.253:4647 [Follower] entering Follower state (Leader: "10.0.31.24:4647") Feb 06 16:59:35 production-nomad-consul-server nomad[2708]: 2018/02/06 16:59:35.133789 [WARN] nomad.heartbeat: node 'f85226c6-69c7-57f3-4b91-9949ee986714' TTL expired Feb 06 17:00:45 production-nomad-consul-server nomad[2708]: 2018/02/06 17:00:45.947467 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "0ed9493e-b65f-2f16-34df-4018496b207e": node is not the leader Feb 06 17:00:46 production-nomad-consul-server nomad[2708]: 2018/02/06 17:00:46.550976 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "a199c208-7511-6840-0977-d78fc7f73e07": node is not the leader Feb 06 17:00:47 production-nomad-consul-server nomad[2708]: 2018/02/06 17:00:47.936533 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "a1586959-eafe-d4ed-9aeb-8446871c609a": node is not the leader Feb 06 17:00:56 production-nomad-consul-server nomad[2708]: 2018/02/06 17:00:56.560357 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "c16252da-2bb4-4181-15f3-fb5f5a896786": node is not the leader Feb 06 17:01:03 production-nomad-consul-server nomad[2708]: 2018/02/06 17:01:03.965246 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "0b2ee85b-0b1f-bbaf-97b5-283080244fac": node is not the leader Feb 06 17:01:06 production-nomad-consul-server nomad[2708]: 2018/02/06 17:01:06.564627 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "37a5d9ee-f3d0-fa4b-922d-c47fa68c2a39": node is not the leader Feb 06 17:01:09 production-nomad-consul-server nomad[2708]: 2018/02/06 17:01:09.522303 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "abce70c5-5668-de7f-37ff-9bd9d296a594": node is not the leader Feb 06 17:01:15 production-nomad-consul-server nomad[2708]: 2018/02/06 17:01:15.131134 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "1c0868dc-8db4-038d-1c75-f4594067892c": node is not the leader Feb 06 17:01:17 production-nomad-consul-server nomad[2708]: 2018/02/06 17:01:17.329867 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "af4a24ce-1aa2-e8e7-4bc0-5937f50af066": node is not the leader Feb 06 17:01:35 production-nomad-consul-server nomad[2708]: 2018/02/06 17:01:35.744618 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "bb420286-0d7b-32ee-5c9c-83caf5e86f8e": node is not the leader Feb 06 17:03:16 production-nomad-consul-server nomad[2708]: 2018/02/06 17:03:16.961910 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "e610256e-98bd-75ee-57c0-ec279e583c1c": node is not the leader Feb 06 17:04:10 production-nomad-consul-server nomad[2708]: 2018/02/06 17:04:10.155017 [WARN] nomad.heartbeat: node '53bb70ea-757e-85fa-5e33-5d95b7d253aa' TTL expired Feb 06 17:04:10 production-nomad-consul-server nomad[2708]: 2018/02/06 17:04:10.155051 [WARN] nomad.heartbeat: node 'f534814b-2a73-ae30-7596-9803ae968747' TTL expired Feb 06 17:05:23 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:23.343212 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "cb8bbbd0-33db-0e3d-2132-015acbf4135a": node is not the leader Feb 06 17:05:29 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:29.858297 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "cb8bbbd0-33db-0e3d-2132-015acbf4135a": node is not the leader Feb 06 17:05:41 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:41.033352 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "954a8a50-1afe-fede-90b8-b33e24d05466": node is not the leader Feb 06 17:05:44 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:44.466499 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "aa2b5c23-ec26-481a-9b3a-5240b53189fe": node is not the leader Feb 06 17:05:47 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:47.453183 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "effe351c-46c1-3cca-71c2-50f647f2b9ef": node is not the leader Feb 06 17:05:47 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:47.829925 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "3b6a731e-4879-0f20-0c2d-69755e5a94f7": node is not the leader Feb 06 17:05:49 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:49.759729 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "e814a873-e924-3b34-a307-ea4b84be75d0": node is not the leader Feb 06 17:05:51 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:51.339366 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "3b6a731e-4879-0f20-0c2d-69755e5a94f7": node is not the leader Feb 06 17:05:53 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:53.938223 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "effe351c-46c1-3cca-71c2-50f647f2b9ef": node is not the leader Feb 06 17:05:55 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:55.114094 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "04d69985-8cdc-cb1c-4137-d0cf2c21a7c5": node is not the leader Feb 06 17:05:58 production-nomad-consul-server nomad[2708]: 2018/02/06 17:05:58.942023 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "e814a873-e924-3b34-a307-ea4b84be75d0": node is not the leader Feb 06 17:06:06 production-nomad-consul-server nomad[2708]: 2018/02/06 17:06:06.536615 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "07556cef-cbb0-49e3-0ad7-3447670bbc3f": node is not the leader Feb 06 17:06:08 production-nomad-consul-server nomad[2708]: 2018/02/06 17:06:08.866128 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "07556cef-cbb0-49e3-0ad7-3447670bbc3f": node is not the leader Feb 06 17:06:14 production-nomad-consul-server nomad[2708]: 2018/02/06 17:06:14.842870 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "fc01880c-0239-4fd3-3fc0-78ee0741c44d": node is not the leader Feb 06 17:06:14 production-nomad-consul-server nomad[2708]: 2018/02/06 17:06:14.842924 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "04d69985-8cdc-cb1c-4137-d0cf2c21a7c5": node is not the leader Feb 06 17:06:33 production-nomad-consul-server nomad[2708]: 2018/02/06 17:06:33.446598 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "8ae2eb86-165f-b420-2de2-68fe33219e29": node is not the leader Feb 06 17:06:47 production-nomad-consul-server nomad[2708]: 2018/02/06 17:06:47.949813 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "a266dd60-1841-60c5-b242-bb2ee7ff6d50": node is not the leader Feb 06 17:08:26 production-nomad-consul-server nomad[2708]: 2018/02/06 17:08:26.938844 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "bc289663-cd4c-77d0-3579-ced92f421f5d": node is not the leader Feb 06 17:08:48 production-nomad-consul-server nomad[2708]: 2018/02/06 17:08:48.470136 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "36146d78-69e6-1e00-9915-7e1975224095": node is not the leader Feb 06 17:09:05 production-nomad-consul-server nomad[2708]: 2018/02/06 17:09:05.940124 [ERR] nomad.deployment_watcher: failed to create evaluation for deployment "36146d78-69e6-1e00-9915-7e1975224095": node is not the leader