2017/04/06 02:19:21 [ERR] yamux: keepalive failed: session shutdown 2017/04/06 02:19:21 [ERR] yamux: Failed to write header: write tcp 10.201.0.8:4647->10.201.7.0:40468: use of closed network connection 2017/04/06 02:49:26 [INFO] raft: Starting snapshot up to 20388972 2017/04/06 02:49:26 [INFO] snapshot: Creating new snapshot at /var/lib/nomad/server/raft/snapshots/3035-20388972-1491446966914.tmp 2017/04/06 02:50:43 [INFO] snapshot: reaping snapshot /var/lib/nomad/server/raft/snapshots/3035-20371975-1491413635162 2017/04/06 02:50:44 [INFO] raft: Compacting logs from 20370125 to 20379624 2017/04/06 02:50:44 [INFO] raft: Snapshot to 20388972 complete 2017/04/06 02:50:54 [WARN] raft: Clearing log suffix from 20390020 to 20390022 2017/04/06 02:50:54 [WARN] raft: Clearing log suffix from 20390023 to 20390083 2017/04/06 02:50:54 [WARN] raft: Failed to get previous log: 20390083 log not found (last: 20390023) 2017/04/06 02:50:54 [ERR] raft-net: Failed to flush response: write tcp 10.201.0.8:4647->10.201.5.6:42114: write: connection reset by peer 2017/04/06 02:54:33.415537 [ERR] http: Request /v1/allocation/fdedb901-d39d-eb50-c83b-835d234cfdc8?index=20393326&wait=30s, error: rpc error: EOF 2017/04/06 02:54:33.415658 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 02:54:33.415765 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 02:54:33.415796 [ERR] http: Request /v1/allocation/fe398a89-f3a7-ee7e-9833-1ea27b6888b1?index=20393620&wait=30s, error: rpc error: EOF 2017/04/06 02:54:33.415880 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 02:54:33.415881 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 02:54:33.436958 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.5.6:4647: getsockopt: connection refused 2017/04/06 02:54:33.436958 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.436961 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.436971 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.517905 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.517929 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.517938 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.517984 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33 [WARN] raft: Rejecting vote request from 10.201.4.6:4647 since we have a leader: 10.201.5.6:4647 2017/04/06 02:54:33 [WARN] raft: Rejecting vote request from 10.201.0.6:4647 since we have a leader: 10.201.5.6:4647 2017/04/06 02:54:33.839281 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.5.6:4647: getsockopt: connection refused 2017/04/06 02:54:33.839295 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.839304 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:33.839316 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 02:54:34 [WARN] raft: Heartbeat timeout from "10.201.5.6:4647" reached, starting election 2017/04/06 02:54:34 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 02:54:34 [ERR] raft: Failed to make RequestVote RPC to 10.201.5.6:4647: EOF 2017/04/06 02:54:35 [INFO] raft: Node at 10.201.0.8:4647 [Follower] entering Follower state (Leader: "") 2017/04/06 02:54:35 [WARN] raft: Rejecting vote request from 10.201.0.7:4647 since we have a leader: 10.201.0.6:4647 2017/04/06 02:54:43 [INFO] memberlist: Suspect ip-10-201-5-6.us-west has failed, no acks received 2017/04/06 02:55:12 [INFO] memberlist: Marking ip-10-201-5-6.us-west as failed, suspect timeout reached (0 peer confirmations) 2017/04/06 02:55:12 [INFO] serf: EventMemberFailed: ip-10-201-5-6.us-west 10.201.5.6 2017/04/06 02:55:12.747567 [INFO] nomad: removing server ip-10-201-5-6.us-west (Addr: 10.201.5.6:4647) (DC: aws-us-west-1) 2017/04/06 02:55:18 [INFO] memberlist: Suspect ip-10-201-5-6.us-west has failed, no acks received 2017/04/06 02:58:14 [INFO] serf: EventMemberJoin: ip-10-201-5-6.us-west 10.201.5.6 2017/04/06 02:58:14.358938 [INFO] nomad: adding server ip-10-201-5-6.us-west (Addr: 10.201.5.6:4647) (DC: aws-us-west-1) 2017/04/06 03:33:11 [INFO] raft: Starting snapshot up to 20397216 2017/04/06 03:33:11 [INFO] snapshot: Creating new snapshot at /var/lib/nomad/server/raft/snapshots/3037-20397216-1491449591138.tmp 2017/04/06 03:34:09 [INFO] snapshot: reaping snapshot /var/lib/nomad/server/raft/snapshots/3035-20380352-1491436182549 2017/04/06 03:34:09 [INFO] raft: Compacting logs from 20379625 to 20386979 2017/04/06 03:34:10 [INFO] raft: Snapshot to 20397216 complete 2017/04/06 03:53:22 [ERR] yamux: keepalive failed: i/o deadline reached 2017/04/06 03:53:22.107502 [ERR] nomad.rpc: multiplex conn accept failed: keepalive timeout 2017/04/06 10:11:51 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 16:19:53 [INFO] raft: Starting snapshot up to 20405496 2017/04/06 16:19:53 [INFO] snapshot: Creating new snapshot at /var/lib/nomad/server/raft/snapshots/3037-20405496-1491495593492.tmp 2017/04/06 16:20:56 [INFO] snapshot: reaping snapshot /var/lib/nomad/server/raft/snapshots/3035-20388972-1491446966914 2017/04/06 16:20:57 [INFO] raft: Compacting logs from 20386980 to 20395264 2017/04/06 16:20:59 [INFO] raft: Snapshot to 20405496 complete 2017/04/06 18:11:47 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 18:41:01 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 18:46:34 [ERR] yamux: keepalive failed: session shutdown 2017/04/06 18:46:34 [ERR] yamux: keepalive failed: session shutdown 2017/04/06 20:15:12 [ERR] yamux: keepalive failed: i/o deadline reached 2017/04/06 20:15:12.491632 [ERR] nomad.rpc: multiplex conn accept failed: keepalive timeout 2017/04/06 21:04:57 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:05:32 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:07:23 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:11:44 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:18:48 [INFO] raft: Starting snapshot up to 20413926 2017/04/06 21:18:48 [INFO] snapshot: Creating new snapshot at /var/lib/nomad/server/raft/snapshots/3037-20413926-1491513528666.tmp 2017/04/06 21:18:56 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:19:55 [INFO] snapshot: reaping snapshot /var/lib/nomad/server/raft/snapshots/3037-20397216-1491449591138 2017/04/06 21:19:56 [INFO] raft: Compacting logs from 20395265 to 20403803 2017/04/06 21:19:57 [INFO] raft: Snapshot to 20413926 complete 2017/04/06 21:23:39 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:30:17 [WARN] raft: Heartbeat timeout from "10.201.0.6:4647" reached, starting election 2017/04/06 21:30:17 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 21:30:18 [INFO] raft: Duplicate RequestVote for same term: 3038 2017/04/06 21:30:18 [INFO] raft: Duplicate RequestVote for same term: 3038 2017/04/06 21:30:18 [ERR] raft-net: Failed to decode incoming command: read tcp 10.201.0.8:4647->10.201.0.6:54814: read: connection reset by peer 2017/04/06 21:30:19 [WARN] raft: Election timeout reached, restarting election 2017/04/06 21:30:19 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 21:30:19 [INFO] raft: Election won. Tally: 3 2017/04/06 21:30:19 [INFO] raft: Node at 10.201.0.8:4647 [Leader] entering Leader state 2017/04/06 21:30:19.536437 [INFO] nomad: cluster leadership acquired 2017/04/06 21:30:19 [INFO] raft: pipelining replication to peer 10.201.5.6:4647 2017/04/06 21:30:19 [INFO] raft: pipelining replication to peer 10.201.4.6:4647 2017/04/06 21:30:19 [INFO] raft: pipelining replication to peer 10.201.0.6:4647 2017/04/06 21:30:19 [INFO] raft: pipelining replication to peer 10.201.0.7:4647 2017/04/06 21:32:19.440826 [WARN] nomad.heartbeat: node '83a65048-c3bb-3403-adea-94c97f7805f8' TTL expired 2017/04/06 21:32:36.523136 [WARN] nomad.heartbeat: node '06fad25e-9188-ceba-ce79-7a8711f4ad84' TTL expired 2017/04/06 21:32:37.967828 [WARN] nomad.heartbeat: node 'cea426e3-9bc7-dae0-7d8f-c4f9ccebee7f' TTL expired 2017/04/06 21:35:21.703944 [WARN] nomad.heartbeat: node '35a550ea-245f-dcb3-368a-14f65da49f88' TTL expired 2017/04/06 21:38:25 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 21:41:24 [WARN] raft: Failed to contact 10.201.4.6:4647 in 500.217345ms 2017/04/06 21:41:24 [WARN] raft: Failed to contact 10.201.4.6:4647 in 956.401467ms 2017/04/06 21:41:25 [WARN] raft: Failed to contact 10.201.4.6:4647 in 1.418480993s 2017/04/06 21:43:40.365960 [WARN] nomad.heartbeat: node '4f77e650-b27d-6d37-1ea5-eea904fdd203' TTL expired 2017/04/06 21:57:23.940142 [WARN] nomad: Attempting to apply large raft entry (type 8) (2816344 bytes) 2017/04/06 22:07:33.062388 [WARN] nomad.heartbeat: node '79ee84a1-1170-44e3-d8ba-38a27e216622' TTL expired 2017/04/06 22:09:48.706995 [WARN] nomad.heartbeat: node 'd229e660-bf1e-23f1-433a-069ff17b815d' TTL expired 2017/04/06 22:11:58.232461 [WARN] nomad.heartbeat: node '273a28ff-5018-5ff6-a2e0-643855a86a3e' TTL expired 2017/04/06 22:12:00.645822 [WARN] nomad.heartbeat: node 'f4150325-e668-42b5-2bda-5011a585f67c' TTL expired 2017/04/06 22:13:21.610334 [WARN] nomad.heartbeat: node 'b53aad75-484c-7758-3422-2e281994e7c6' TTL expired 2017/04/06 22:13:21.631242 [WARN] nomad.heartbeat: node '71b5d369-ba5c-cfdf-8298-088903206322' TTL expired 2017/04/06 22:13:21.797812 [WARN] nomad.heartbeat: node '7f14f02b-adba-d564-f121-9465e52b4eab' TTL expired 2017/04/06 22:13:22.518750 [WARN] nomad.heartbeat: node '6e7c39cc-a76b-89f5-c312-675bebdbf7c4' TTL expired 2017/04/06 22:13:22.736021 [WARN] nomad.heartbeat: node '2fb2fb98-38aa-3e85-f7c5-06bdd462e360' TTL expired 2017/04/06 22:13:24.086110 [WARN] nomad.heartbeat: node '08158fc4-a197-e516-340c-9e64e5e7cbc8' TTL expired 2017/04/06 22:13:24.384246 [WARN] nomad.heartbeat: node 'c1b72a3f-d5ba-52f0-8799-1fe747fc5b88' TTL expired 2017/04/06 22:13:24.766867 [WARN] nomad.heartbeat: node '6d39bfa7-cf9b-96f2-0e60-0e582b693f29' TTL expired 2017/04/06 22:13:26.898708 [WARN] nomad.heartbeat: node 'de51dea1-83ba-ea33-2a28-c64aaf585143' TTL expired 2017/04/06 22:13:27.528067 [WARN] nomad.heartbeat: node '1b68666c-8b61-2261-0e0e-3e42ad77c7ca' TTL expired 2017/04/06 22:13:29.875288 [WARN] nomad.heartbeat: node 'b0a2bb93-ea25-c75c-6f16-aafaee531377' TTL expired 2017/04/06 22:13:33.928463 [WARN] nomad.heartbeat: node '29779952-9695-5f3f-e4b0-d961ecfbd5c7' TTL expired 2017/04/06 22:13:34.833912 [WARN] nomad.heartbeat: node 'bb4a5e3c-ae61-04a7-d00e-bdaa44047a0e' TTL expired 2017/04/06 22:13:35.190510 [WARN] nomad.heartbeat: node 'fd92415c-5f2c-ec46-1f03-333acb533602' TTL expired 2017/04/06 22:13:35.570646 [WARN] nomad.heartbeat: node 'dd5dd0ed-d408-03fb-f51f-d40ac7559116' TTL expired 2017/04/06 22:13:36.543152 [WARN] nomad.heartbeat: node 'b04fb230-34d7-0fea-bfa7-ddf7703c3280' TTL expired 2017/04/06 22:13:38.258488 [WARN] nomad.heartbeat: node '1816256b-949b-7be3-523f-5a22ea9e6a54' TTL expired 2017/04/06 22:13:39.719855 [WARN] nomad.heartbeat: node '2e0953cb-b536-fdba-d144-c4f02f2522b0' TTL expired 2017/04/06 22:13:40.265260 [WARN] nomad.heartbeat: node 'eb035825-2a8f-403c-5150-36531e512646' TTL expired 2017/04/06 22:13:40.441615 [WARN] nomad.heartbeat: node 'ed4a878c-9433-0d2f-1aa9-00c4d4dba533' TTL expired 2017/04/06 22:13:40.793330 [WARN] nomad.heartbeat: node '47cc7427-190c-4f4f-4efc-6e904508cb11' TTL expired 2017/04/06 22:13:42.714253 [WARN] nomad.heartbeat: node '2a39a420-0a45-9305-ad59-e318c0493246' TTL expired 2017/04/06 22:13:43.175314 [WARN] nomad.heartbeat: node '5a481c42-136f-e4bf-c410-57c5f8205031' TTL expired 2017/04/06 22:13:45.106055 [WARN] nomad.heartbeat: node '66cb6fed-c0f2-b4a7-b310-75cecc8218a3' TTL expired 2017/04/06 22:13:46.597583 [WARN] nomad.heartbeat: node 'aa8b407f-b015-ad3f-ecba-6de313d2c052' TTL expired 2017/04/06 22:13:47.471110 [WARN] nomad.heartbeat: node 'ddec4606-3dc4-b364-eb7a-acb78f791e50' TTL expired 2017/04/06 22:13:49.389628 [WARN] nomad.heartbeat: node '51ec9c91-5fa3-2dfa-3f86-d1efe8b19dd5' TTL expired 2017/04/06 22:13:50.403246 [WARN] nomad.heartbeat: node '7af45de0-6f78-a0b3-dcd3-135c81bf7c39' TTL expired 2017/04/06 22:13:51.480111 [WARN] nomad.heartbeat: node '5b2b2df6-b150-0b19-75bd-a464a35e569d' TTL expired 2017/04/06 22:14:57.000692 [WARN] nomad.heartbeat: node 'a3952289-5b96-b5dd-934c-8e63ac965287' TTL expired 2017/04/06 22:14:57.053178 [WARN] nomad.heartbeat: node 'c71f9f62-4127-13ac-84ba-10e3b69d9528' TTL expired 2017/04/06 22:15:01.197669 [WARN] nomad.heartbeat: node '92c2d959-27af-dbf6-8445-dc76cf5481ff' TTL expired 2017/04/06 22:15:01.217839 [WARN] nomad.heartbeat: node '46e6a276-e874-0ccc-4217-a90bae7bae4d' TTL expired 2017/04/06 22:15:02.463330 [WARN] nomad.heartbeat: node 'e2af54a6-1179-a187-2d47-fe36b912da38' TTL expired 2017/04/06 22:15:02.924608 [WARN] nomad.heartbeat: node '5a1cbcea-0456-0b51-957e-3e3e4dbf577f' TTL expired 2017/04/06 22:15:04.610208 [WARN] nomad.heartbeat: node 'ab6b9cd0-a0e4-d734-71f0-578829657fb7' TTL expired 2017/04/06 22:15:05.187000 [WARN] nomad.heartbeat: node '11b49cba-51e1-31e3-f6fa-85dbe5932f78' TTL expired 2017/04/06 22:15:08.389428 [WARN] nomad.heartbeat: node '32d1c367-0cb5-3cdd-883c-dd38a4956e0e' TTL expired 2017/04/06 22:15:11.271047 [WARN] nomad.heartbeat: node '38137f2f-d543-34c3-c711-a6233bacc08b' TTL expired 2017/04/06 22:15:12.896661 [WARN] nomad.heartbeat: node '066a003a-a352-ad0e-c05c-336d2655ea16' TTL expired 2017/04/06 22:15:13.675130 [WARN] nomad.heartbeat: node '9d7143e3-c342-8adc-300a-0a55dc9efa44' TTL expired 2017/04/06 22:15:14.166610 [WARN] nomad.heartbeat: node '0ebcfe08-daa3-e92b-664a-593141f84547' TTL expired 2017/04/06 22:15:19.524450 [WARN] nomad.heartbeat: node '4087efc2-d6cb-1fb5-872d-6e1f97525551' TTL expired 2017/04/06 22:15:22.546398 [WARN] nomad.heartbeat: node 'd64f6d7a-0484-6ee4-e7a5-81613d13955b' TTL expired 2017/04/06 22:15:30.593189 [WARN] nomad.heartbeat: node 'aba3c042-960c-baaa-6f4a-ea3ff894d612' TTL expired 2017/04/06 22:16:58.100473 [WARN] nomad.heartbeat: node '616d6d13-57b2-2148-fc53-9e2db4b452bb' TTL expired 2017/04/06 22:17:16.921234 [WARN] nomad.heartbeat: node '5c2fcd74-4da6-d18e-3d60-01ff3eb75567' TTL expired 2017/04/06 22:17:29.761253 [WARN] nomad.heartbeat: node 'b6d789b2-3721-8bd1-65b1-63c59b6aa65e' TTL expired 2017/04/06 22:30:43 [ERR] yamux: keepalive failed: session shutdown 2017/04/06 22:30:45.693103 [WARN] nomad: Attempting to apply large raft entry (type 8) (1901802 bytes) 2017/04/06 22:30:53.178886 [WARN] nomad.heartbeat: node 'a8a7f3e1-6feb-7cd0-43a1-5cd2520c4afc' TTL expired 2017/04/06 22:30:53.235330 [WARN] nomad.heartbeat: node '05af79a4-2368-50c2-2567-e60395387d24' TTL expired 2017/04/06 22:30:53.585899 [WARN] nomad.heartbeat: node 'b4f83013-c647-05bc-1bb6-28c5f09fc325' TTL expired 2017/04/06 22:30:53.729962 [WARN] nomad.heartbeat: node '60bf5a7a-0394-a840-287c-35731c56a25b' TTL expired 2017/04/06 22:30:53.930078 [WARN] nomad.heartbeat: node '234492ef-15b5-c6ec-9fa5-b47129744451' TTL expired 2017/04/06 22:30:53.944579 [WARN] nomad.heartbeat: node '15c7370e-a0f1-62f8-ef4c-a208f7804294' TTL expired 2017/04/06 22:30:54.063102 [WARN] nomad.heartbeat: node 'fcfe0b93-13a7-a968-9cef-26f9bfde37ed' TTL expired 2017/04/06 22:30:54.063202 [WARN] nomad.heartbeat: node '16125d8b-850f-1f0f-0553-b2a34500fdb2' TTL expired 2017/04/06 22:30:54.071245 [WARN] nomad.heartbeat: node '67523a01-c3cf-9aec-fd37-fa4f58976df4' TTL expired 2017/04/06 22:30:54.290468 [WARN] nomad.heartbeat: node 'd04d2345-3d4f-9215-5a8e-270630e3152e' TTL expired 2017/04/06 22:30:54.302485 [WARN] nomad.heartbeat: node '2b0324fb-9844-e9d3-ae1c-b11f6446ba88' TTL expired 2017/04/06 22:30:54.429047 [WARN] nomad.heartbeat: node '48fdb6b9-e937-df00-b4db-6f79e90aa5ab' TTL expired 2017/04/06 22:30:54.503151 [WARN] nomad.heartbeat: node 'f1472bd5-3e85-72f9-4bcc-5a551feb003a' TTL expired 2017/04/06 22:30:54.538373 [WARN] nomad.heartbeat: node '512408c8-5d46-44e5-fe55-8e89572059ce' TTL expired 2017/04/06 22:30:54.613465 [WARN] nomad.heartbeat: node 'fd125f94-cccc-58d8-e859-7e59f50378f3' TTL expired 2017/04/06 22:30:54.653630 [WARN] nomad.heartbeat: node 'ac47d7d9-8235-20eb-71f1-1e25676cee57' TTL expired 2017/04/06 22:30:54.701186 [WARN] nomad.heartbeat: node 'aadad202-c46c-32d3-9f72-c3dee309b59a' TTL expired 2017/04/06 22:30:54.729803 [WARN] nomad.heartbeat: node '08bd1096-bcbb-ac56-7f36-82a25623844c' TTL expired 2017/04/06 22:30:54.864425 [WARN] nomad.heartbeat: node '4ee9cfab-80d0-cf77-2893-89511b540770' TTL expired 2017/04/06 22:30:54.965615 [WARN] nomad.heartbeat: node '4b7a564d-efeb-3d5d-3760-461b943bb295' TTL expired 2017/04/06 22:30:55.015698 [WARN] nomad.heartbeat: node 'd30c1590-9275-7674-0fee-aa6b5b365376' TTL expired 2017/04/06 22:30:55.124711 [WARN] nomad.heartbeat: node '9f3dd969-ac55-ae5e-aca0-aec893614e57' TTL expired 2017/04/06 22:30:55.132711 [WARN] nomad.heartbeat: node '124696f3-acf4-bdf7-48ab-492fa1825153' TTL expired 2017/04/06 22:30:55.138198 [WARN] nomad.heartbeat: node '3cf45169-ccbb-18cc-8a93-842d3ebf7755' TTL expired 2017/04/06 22:30:55.219883 [WARN] nomad.heartbeat: node '44fb8cdf-0257-c981-d33f-a1cc4ea9c68d' TTL expired 2017/04/06 22:30:55.242179 [WARN] nomad.heartbeat: node '70a8c64f-af14-5762-b3a1-2f581e607aeb' TTL expired 2017/04/06 22:30:55.277208 [WARN] nomad.heartbeat: node 'a1856ef3-bdda-4779-9e40-36b9fb40fc49' TTL expired 2017/04/06 22:30:55.324403 [WARN] nomad.heartbeat: node '84981796-07b7-a2d7-cec5-bffddafc0169' TTL expired 2017/04/06 22:30:55.392525 [WARN] nomad.heartbeat: node 'e2f44170-70af-adfd-00d7-418af8a033cb' TTL expired 2017/04/06 22:30:55.460281 [WARN] nomad.heartbeat: node '3389c785-9bc2-5b2f-615c-c0741d5fbfee' TTL expired 2017/04/06 22:30:55.511008 [WARN] nomad.heartbeat: node 'd47e2424-58bc-ac87-9bda-8118f37d5292' TTL expired 2017/04/06 22:30:55.687231 [WARN] nomad.heartbeat: node 'ef3f9bef-b896-6b71-c41e-5d35f782d562' TTL expired 2017/04/06 22:30:55.741692 [WARN] nomad.heartbeat: node '708e1847-f5e5-0302-e69b-e5746d86eb07' TTL expired 2017/04/06 22:30:55.798144 [WARN] nomad.heartbeat: node 'e6743093-6a03-b8ab-b796-f674f506a733' TTL expired 2017/04/06 22:30:55.848095 [WARN] nomad.heartbeat: node '67f34da0-26ba-d85b-7caf-967f02ca0b1d' TTL expired 2017/04/06 22:30:55.866289 [WARN] nomad.heartbeat: node '51829a62-6b5a-be8e-70fd-2bf163f9aed2' TTL expired 2017/04/06 22:30:55.869489 [WARN] nomad.heartbeat: node '760cce8c-7d91-0db9-5153-bb0b7f59f2f1' TTL expired 2017/04/06 22:30:55.894405 [WARN] nomad.heartbeat: node 'd28eddfa-daa9-f3d5-b0bf-786f777b490c' TTL expired 2017/04/06 22:30:55.969140 [WARN] nomad.heartbeat: node 'b083814e-9fbd-5a16-4466-af2cf702b32b' TTL expired 2017/04/06 22:30:56.090171 [WARN] nomad.heartbeat: node 'dc66d424-62fb-b375-e972-adb69ac8ed7a' TTL expired 2017/04/06 22:30:56.125191 [WARN] nomad.heartbeat: node '3a37b013-d073-fdcb-62d5-da42f2d929d1' TTL expired 2017/04/06 22:30:56.125306 [WARN] nomad.heartbeat: node 'c25a5bb4-16f1-47eb-264d-7ef1de8fa6ae' TTL expired 2017/04/06 22:30:56.125604 [WARN] nomad.heartbeat: node 'b280b87a-a16e-9d68-6748-1c107bf575ba' TTL expired 2017/04/06 22:30:56.177245 [WARN] nomad.heartbeat: node 'da5fd28e-d7ed-f163-25a7-827b064f6b81' TTL expired 2017/04/06 22:30:56.199045 [WARN] nomad.heartbeat: node '85d25ff0-ecc2-db99-e7f5-6994987117b8' TTL expired 2017/04/06 22:30:56.495875 [WARN] nomad.heartbeat: node '633c165f-a1cb-0812-9d52-ea16d62162dc' TTL expired 2017/04/06 22:30:56.560524 [WARN] nomad.heartbeat: node 'b8e8b551-f635-e9fc-9dd7-7b855d38ffcb' TTL expired 2017/04/06 22:30:56.607051 [WARN] nomad.heartbeat: node '287262b8-6cc7-bce7-3aff-342619b71950' TTL expired 2017/04/06 22:30:56.697222 [WARN] nomad.heartbeat: node 'df5a8b19-464e-7bd5-b086-2af1f342d809' TTL expired 2017/04/06 22:30:56.948813 [WARN] nomad.heartbeat: node '69c5ec88-47ee-a610-aa4f-1eb3cf6dea0d' TTL expired 2017/04/06 22:30:57.259159 [WARN] nomad.heartbeat: node '6ebfc8b0-f90c-75db-1002-a92bd69dfb54' TTL expired 2017/04/06 22:30:57.278971 [WARN] nomad.heartbeat: node '9fd93659-fa01-5bca-486c-0281602e9473' TTL expired 2017/04/06 22:30:57.324301 [WARN] nomad.heartbeat: node '1d2fc9ff-1055-1dbb-0d49-88ae774abf94' TTL expired 2017/04/06 22:30:57.381202 [WARN] nomad.heartbeat: node 'c03a47d2-91ca-ee76-fe4c-aa395cab4444' TTL expired 2017/04/06 22:30:57.659365 [WARN] nomad.heartbeat: node '4e2e5325-b865-8dd9-8dd2-ed76aaef411e' TTL expired 2017/04/06 22:30:57.663515 [WARN] nomad.heartbeat: node 'ec7ad106-5ceb-e934-3447-1d8278ae0145' TTL expired 2017/04/06 22:30:57.984573 [WARN] nomad.heartbeat: node '73cc85a9-0866-f79b-40c4-fac964e2f002' TTL expired 2017/04/06 22:30:58.057025 [WARN] nomad.heartbeat: node 'b552e586-d642-9a90-98d5-85b0000e0073' TTL expired 2017/04/06 22:30:58.099857 [WARN] nomad.heartbeat: node 'b2df7482-e045-49e3-b554-79fdeadadcda' TTL expired 2017/04/06 22:30:58.106036 [WARN] nomad.heartbeat: node 'cead8a94-ab73-fbe2-4f4c-daa1323f2644' TTL expired 2017/04/06 22:30:58.122207 [WARN] nomad.heartbeat: node '1d44b110-f816-a5ac-51e1-a3eeadebeb34' TTL expired 2017/04/06 22:30:58.149037 [WARN] nomad.heartbeat: node '2bea1876-cae9-67e6-001b-15584c055b55' TTL expired 2017/04/06 22:30:58.237603 [WARN] nomad.heartbeat: node 'a79237dc-a2e4-efb1-fc6a-b88cb4a8c208' TTL expired 2017/04/06 22:30:58.323903 [WARN] nomad.heartbeat: node '48d98500-426b-4422-2c15-c571cd593ddc' TTL expired 2017/04/06 22:30:58.357850 [WARN] nomad.heartbeat: node '999f2841-1f49-f381-d48e-553ca8461832' TTL expired 2017/04/06 22:30:58.405673 [WARN] nomad.heartbeat: node '2d31aa40-4d22-d2f1-0206-59fecef55b79' TTL expired 2017/04/06 22:30:58.470122 [WARN] nomad.heartbeat: node 'ebd76181-0cef-b011-a9e4-f71fedbc8374' TTL expired 2017/04/06 22:30:58.525117 [WARN] nomad.heartbeat: node 'f62efd7a-8097-674a-82ef-94656ab0dcfe' TTL expired 2017/04/06 22:30:58.747042 [WARN] nomad.heartbeat: node '49ed0285-abd0-3a7d-04bd-d81a3c5cef5a' TTL expired 2017/04/06 22:30:58.838310 [WARN] nomad.heartbeat: node '724e4da8-5bf1-7fab-c78b-d873046b7f2b' TTL expired 2017/04/06 22:30:58.851010 [WARN] nomad.heartbeat: node '5e0b628f-8992-4f76-8d81-e4bc53cb9dc4' TTL expired 2017/04/06 22:30:58.910306 [WARN] nomad.heartbeat: node 'caf2d4bf-8f1d-7b44-3ad9-0fd937f93e31' TTL expired 2017/04/06 22:30:58.937449 [WARN] nomad.heartbeat: node '09c354a1-6025-d2da-7fa9-7d3eb2cd0b5a' TTL expired 2017/04/06 22:30:58.943609 [WARN] nomad.heartbeat: node '3650ad94-0a65-dd9b-9937-e48f8d9867d1' TTL expired 2017/04/06 22:30:58.978756 [WARN] nomad.heartbeat: node '1a1d0606-21b8-66af-dd42-10cd9d7026d9' TTL expired 2017/04/06 22:30:58.986022 [WARN] nomad.heartbeat: node '5f9de81f-1e40-cb9e-4227-06ba1295ebfc' TTL expired 2017/04/06 22:30:59.001129 [WARN] nomad.heartbeat: node '60407e44-defd-dcb4-496b-2a4bcff23cb4' TTL expired 2017/04/06 22:30:59.107841 [WARN] nomad.heartbeat: node '0044ff80-2b53-df11-b6d5-15d810313694' TTL expired 2017/04/06 22:30:59.126502 [WARN] nomad.heartbeat: node 'c9b49430-3d6d-72ac-3b06-2dc453f4561d' TTL expired 2017/04/06 22:30:59.133647 [WARN] nomad.heartbeat: node 'd7ca3ca6-2aac-cec5-9e21-a46acd3369dc' TTL expired 2017/04/06 22:30:59.153294 [WARN] nomad.heartbeat: node 'a9caa4dc-c45a-e165-be2b-17f746cffccd' TTL expired 2017/04/06 22:30:59.515397 [WARN] nomad.heartbeat: node 'b97df0a8-64e7-d392-c8cf-2c4f84a122a9' TTL expired 2017/04/06 22:30:59.597267 [WARN] nomad.heartbeat: node '84242775-cfda-9d8e-ec1a-a3c7e810c2c4' TTL expired 2017/04/06 22:30:59.711447 [WARN] nomad.heartbeat: node 'ba504d1e-a753-aec5-f6f5-fdf527a060a3' TTL expired 2017/04/06 22:30:59.727955 [WARN] nomad.heartbeat: node 'a2e9befb-64e6-838c-2730-29ee0e70ccb6' TTL expired 2017/04/06 22:30:59.973975 [WARN] nomad.heartbeat: node '2bcad5a4-e8ff-1511-f290-ab0674b88c64' TTL expired 2017/04/06 22:30:59.981996 [WARN] nomad.heartbeat: node 'eb5b4a4f-ca23-384e-c85f-0b06146ce96b' TTL expired 2017/04/06 22:31:00.099247 [WARN] nomad.heartbeat: node '96fd9ad9-5682-4145-c07e-8200eb4424d1' TTL expired 2017/04/06 22:31:00.267396 [WARN] nomad.heartbeat: node 'a141c30f-0b4e-d26a-2813-59022c050362' TTL expired 2017/04/06 22:31:00.289201 [WARN] nomad.heartbeat: node '4704ce58-ff7b-950f-60b9-da775c73a4cd' TTL expired 2017/04/06 22:31:00.342905 [WARN] nomad.heartbeat: node 'e45f592b-c26b-cee2-8a43-80de98045440' TTL expired 2017/04/06 22:31:00.402527 [WARN] nomad.heartbeat: node 'b5475c1f-a1d1-4405-0492-76be6053b2e7' TTL expired 2017/04/06 22:31:00.410325 [WARN] nomad.heartbeat: node 'f2962959-725b-dd3d-2c13-8d9579d9991b' TTL expired 2017/04/06 22:31:00.432858 [WARN] nomad.heartbeat: node 'd3a207cb-9315-5fd7-c87c-8e273685966d' TTL expired 2017/04/06 22:31:00.518526 [WARN] nomad.heartbeat: node '9a7b45c1-dc0a-fcec-e865-d0ec229edf76' TTL expired 2017/04/06 22:31:00.546847 [WARN] nomad.heartbeat: node '5ad7ad90-4ed1-a7c4-b18a-ef81f1a18141' TTL expired 2017/04/06 22:31:00.566712 [WARN] nomad.heartbeat: node 'fdec4dab-a4e2-d0b1-0205-65c154b9f8c2' TTL expired 2017/04/06 22:31:00.601798 [WARN] nomad.heartbeat: node '4934ed54-b9cb-39c9-20f2-1d5fbc58aa85' TTL expired 2017/04/06 22:31:00.646874 [WARN] nomad.heartbeat: node 'a6debf12-30fc-3ff2-dea5-1c32084b2ab2' TTL expired 2017/04/06 22:31:00.785165 [WARN] nomad.heartbeat: node 'c8712c53-ed42-b4d6-a259-2c02529545ce' TTL expired 2017/04/06 22:31:00.878496 [WARN] nomad.heartbeat: node '2f86cc1e-383b-8dd8-a6c8-7ed02c511b68' TTL expired 2017/04/06 22:31:00.938218 [WARN] nomad.heartbeat: node 'b685de1e-17c1-0788-e8d0-8dcec20b1f90' TTL expired 2017/04/06 22:31:00.951801 [WARN] nomad.heartbeat: node '568d26a1-8db0-455a-9fd8-aab4f25b6d44' TTL expired 2017/04/06 22:31:01.032163 [WARN] nomad.heartbeat: node '868833c6-2277-cc70-d994-b9365f4417c6' TTL expired 2017/04/06 22:31:01.154098 [WARN] nomad.heartbeat: node '51d405ef-ea31-a421-f679-457dbf3e64d7' TTL expired 2017/04/06 22:31:01.168687 [WARN] nomad.heartbeat: node '3b8bf6a0-7b83-f8e2-ba57-7b9bfbe7d13a' TTL expired 2017/04/06 22:31:01.176046 [WARN] nomad.heartbeat: node '2f3663aa-5b3f-ba6e-edd4-8b07ce2ba37e' TTL expired 2017/04/06 22:31:01.192564 [WARN] nomad.heartbeat: node 'bf88ae01-f5df-ccef-e9a7-39353b87cd61' TTL expired 2017/04/06 22:31:01.341935 [WARN] nomad.heartbeat: node '816ec447-bfa8-5d35-c667-c4f0b96dba14' TTL expired 2017/04/06 22:31:01.514540 [WARN] nomad.heartbeat: node 'b5ca3d30-8ea9-e71b-3381-c51042784818' TTL expired 2017/04/06 22:31:01.577798 [WARN] nomad.heartbeat: node '78ef98f9-1370-41f9-9ee9-d427042b54c7' TTL expired 2017/04/06 22:31:01.595150 [WARN] nomad.heartbeat: node '0503e30c-df9c-9149-ce6a-62fe98634206' TTL expired 2017/04/06 22:31:01.733864 [WARN] nomad.heartbeat: node '028e4844-b458-0adc-b567-1ee5cbc06e3b' TTL expired 2017/04/06 22:31:01.796703 [WARN] nomad.heartbeat: node '38cd8b41-29f8-cec8-7a9e-3d0b3a6e6fd8' TTL expired 2017/04/06 22:31:01.852638 [WARN] nomad.heartbeat: node '509dd77f-eee9-f473-c454-e497555861ac' TTL expired 2017/04/06 22:31:01.907633 [WARN] nomad.heartbeat: node 'be4bcf14-27fa-be12-cade-575c9cf12ef6' TTL expired 2017/04/06 22:31:01.935323 [WARN] nomad.heartbeat: node '3c6302df-9e29-eb36-5346-56f23c572829' TTL expired 2017/04/06 22:31:02.011478 [WARN] nomad.heartbeat: node 'e04b66e4-8af4-244d-2791-d9d3d4b39a53' TTL expired 2017/04/06 22:31:02.171738 [WARN] nomad.heartbeat: node '6a2eddda-ac74-0e00-8d38-95f4e623f14f' TTL expired 2017/04/06 22:31:02.370508 [WARN] nomad.heartbeat: node 'b4d71057-2250-c28f-2f77-951998d46148' TTL expired 2017/04/06 22:31:02.403741 [WARN] nomad.heartbeat: node '22e762e5-a843-5145-c635-1fc66b16a194' TTL expired 2017/04/06 22:31:02.514474 [WARN] nomad.heartbeat: node '0b9449e5-05f3-a5c0-d685-7c18992d1915' TTL expired 2017/04/06 22:31:02.652196 [WARN] nomad.heartbeat: node 'a3b12d5a-bc78-3a10-1ea0-789ee682fe55' TTL expired 2017/04/06 22:31:02.698436 [WARN] nomad.heartbeat: node 'ea61a715-481f-633e-65ea-8ef057c7e355' TTL expired 2017/04/06 22:31:02.753196 [WARN] nomad.heartbeat: node 'a5bec20a-79c2-8521-5af1-ae347297f11b' TTL expired 2017/04/06 22:31:02.798961 [WARN] nomad.heartbeat: node '6a1bf5c2-e2dd-a9c7-d539-994226934625' TTL expired 2017/04/06 22:31:02.902204 [WARN] nomad.heartbeat: node 'b1932536-ebf6-5dc0-6471-244c87d4dc20' TTL expired 2017/04/06 22:31:02.969024 [WARN] nomad.heartbeat: node '58f0788c-e2b6-58c3-1d84-4537cb91a418' TTL expired 2017/04/06 22:31:02.976770 [WARN] nomad.heartbeat: node 'c917de8f-da20-35b7-e97d-9b69f9f63f89' TTL expired 2017/04/06 22:31:03.248178 [WARN] nomad.heartbeat: node '286e29e5-a984-260b-c947-c4e3768cbbf9' TTL expired 2017/04/06 22:31:03.258330 [WARN] nomad.heartbeat: node '6cc5a79e-bc12-76de-885a-c641fd8f302f' TTL expired 2017/04/06 22:31:03.292262 [WARN] nomad.heartbeat: node '193b8172-8388-06a8-01e0-dd7009ddacf3' TTL expired 2017/04/06 22:31:03.418650 [WARN] nomad.heartbeat: node '6aea6fb6-3fff-2a3d-6cd1-791d1a4a1580' TTL expired 2017/04/06 22:31:03.464790 [WARN] nomad.heartbeat: node 'abc0b1b6-35a0-401e-4bb9-3b5efc90e911' TTL expired 2017/04/06 22:31:03.491776 [WARN] nomad.heartbeat: node '3745f422-7c82-b063-844f-54e54fe4aadd' TTL expired 2017/04/06 22:31:03.564755 [WARN] nomad.heartbeat: node '009e36d0-4cc3-9616-b6ed-fdaeb353863b' TTL expired 2017/04/06 22:31:03.568188 [WARN] nomad.heartbeat: node 'db87311f-83fe-5913-7968-f8c3952f86ad' TTL expired 2017/04/06 22:31:03.639052 [WARN] nomad.heartbeat: node 'cfa143c2-d01e-a34b-7188-20314e0e3b07' TTL expired 2017/04/06 22:31:03.787383 [WARN] nomad.heartbeat: node '1cfed909-fd9e-893f-5579-964f7c46cf8d' TTL expired 2017/04/06 22:31:04.001849 [WARN] nomad.heartbeat: node 'a2778527-70b8-7ff8-a6d6-08f2009803d1' TTL expired 2017/04/06 22:31:04.114186 [WARN] nomad.heartbeat: node 'ff17d2e3-9f48-beb2-563e-3a2c0581e28c' TTL expired 2017/04/06 22:31:04.299386 [WARN] nomad.heartbeat: node 'fd4d8fde-ab9d-19f8-464b-a7cd88ac10bf' TTL expired 2017/04/06 22:31:04.394019 [WARN] nomad.heartbeat: node '8c071964-bbeb-610f-f682-dabb8702adba' TTL expired 2017/04/06 22:31:04.411297 [WARN] nomad.heartbeat: node 'bc179276-fe09-c3bb-ca7d-b238bfaf20fe' TTL expired 2017/04/06 22:31:04.429973 [WARN] nomad.heartbeat: node '7b39a7a2-ebca-f349-8bd9-db86fdc56876' TTL expired 2017/04/06 22:31:04.661586 [WARN] nomad.heartbeat: node '05020ac8-ffa4-6528-1caa-861eb8c760f2' TTL expired 2017/04/06 22:31:04.749032 [WARN] nomad.heartbeat: node '1356e8d9-cc5c-e1a7-87c6-318020f1fc30' TTL expired 2017/04/06 22:31:04.780674 [WARN] nomad.heartbeat: node '0f1a7cd3-f618-1b2d-c5ee-4493b6b05555' TTL expired 2017/04/06 22:31:04.784567 [WARN] nomad.heartbeat: node '4bfec390-6c94-9d64-93ce-33c243c7f379' TTL expired 2017/04/06 22:31:04.881098 [WARN] nomad.heartbeat: node 'ad96aead-03ff-34b1-b44c-deea7c4cad87' TTL expired 2017/04/06 22:31:04.897795 [WARN] nomad.heartbeat: node '119c3b21-2912-5ac3-dba9-06a4d58d68a9' TTL expired 2017/04/06 22:31:04.930960 [WARN] nomad.heartbeat: node 'fd47ea1a-805d-a9c8-b5ea-f352cf79fa8e' TTL expired 2017/04/06 22:31:05.017381 [WARN] nomad.heartbeat: node '895c6fa5-7e0c-de08-84ba-5d4053dcf6ef' TTL expired 2017/04/06 22:31:05.018843 [WARN] nomad.heartbeat: node '241c5d7b-d000-6e7b-57f8-13fbc666cc6d' TTL expired 2017/04/06 22:31:05.075125 [WARN] nomad.heartbeat: node 'c27d1443-7d04-1545-f17e-97afb714be02' TTL expired 2017/04/06 22:31:05.236579 [WARN] nomad.heartbeat: node '85f46f14-5e24-f111-e689-df55d0004a37' TTL expired 2017/04/06 22:31:05.283348 [WARN] nomad.heartbeat: node '6d2fcfab-b470-376b-13f5-ab52ac382168' TTL expired 2017/04/06 22:31:05.373875 [WARN] nomad.heartbeat: node '32cce0ef-dae8-8d0b-c222-fb4e5a23bc5d' TTL expired 2017/04/06 22:31:05.437812 [WARN] nomad.heartbeat: node '80792911-8f81-63db-aa65-e90200deb667' TTL expired 2017/04/06 22:31:05.558431 [WARN] nomad.heartbeat: node '4a906aaa-84b2-7c6c-cb0c-da3112a25965' TTL expired 2017/04/06 22:31:05.562825 [WARN] nomad.heartbeat: node '5ae894ac-86df-89b0-cbdd-72a8ddd4b268' TTL expired 2017/04/06 22:31:05.596533 [WARN] nomad.heartbeat: node 'e0015ae5-e8fe-fc80-3553-6f5d12fe988c' TTL expired 2017/04/06 22:31:05.766893 [WARN] nomad.heartbeat: node '3d798652-59a9-f684-5c19-14df5c0e35a5' TTL expired 2017/04/06 22:31:05.770430 [WARN] nomad.heartbeat: node 'b5503fa4-9a54-42ce-a2ab-7360c4612a6c' TTL expired 2017/04/06 22:31:05.770431 [WARN] nomad.heartbeat: node 'a8d3a7eb-2be2-5bf7-27be-46b42323538c' TTL expired 2017/04/06 22:31:05.771329 [WARN] nomad.heartbeat: node 'fd08c5a2-31d8-0475-3381-8af41f058208' TTL expired 2017/04/06 22:31:05.780463 [WARN] nomad.heartbeat: node '856a59dc-9ea3-832e-32f9-0c151db655ce' TTL expired 2017/04/06 22:31:05.957155 [WARN] nomad.heartbeat: node 'a9e63302-17e9-a851-5497-130b990c3a02' TTL expired 2017/04/06 22:31:06.087558 [WARN] nomad.heartbeat: node '7d9d8b45-51e4-fe58-8ffc-667d32e8d2e2' TTL expired 2017/04/06 22:31:06.118499 [WARN] nomad.heartbeat: node 'dbf9f4fd-b39f-d53f-a240-6ce5f248decb' TTL expired 2017/04/06 22:31:06.120604 [WARN] nomad.heartbeat: node 'dfb41106-ad81-c1e1-0496-5954463b2610' TTL expired 2017/04/06 22:31:06.149804 [WARN] nomad.heartbeat: node 'cac5760b-1fcb-384d-2558-e4a880037e1a' TTL expired 2017/04/06 22:31:06.172760 [WARN] nomad.heartbeat: node 'f5eec69e-b3b7-c0b5-7351-e64da20d3414' TTL expired 2017/04/06 22:31:06.253901 [WARN] nomad.heartbeat: node '72369af6-e0de-9784-cad6-cabafafced90' TTL expired 2017/04/06 22:31:06.264372 [WARN] nomad.heartbeat: node '4380261d-7766-bc27-96b7-9bafce433cb0' TTL expired 2017/04/06 22:31:06.458846 [WARN] nomad.heartbeat: node '8d14a82a-def9-ba19-91f8-22fe667154ba' TTL expired 2017/04/06 22:31:06.482328 [WARN] nomad.heartbeat: node '4f8290ff-c121-178d-416d-fa6491f751c8' TTL expired 2017/04/06 22:31:06.484155 [WARN] nomad.heartbeat: node '33a62cb2-4d6c-966c-6651-f6fd926539ef' TTL expired 2017/04/06 22:31:06.489997 [WARN] nomad.heartbeat: node '76ad8a04-494c-7445-4d78-28e1cfab0455' TTL expired 2017/04/06 22:31:06.670922 [WARN] nomad.heartbeat: node '040764a7-032a-0c77-73c3-0db76975a745' TTL expired 2017/04/06 22:31:06.713718 [WARN] nomad.heartbeat: node '6b852e12-4568-2e56-c3e3-19a9f8c77e78' TTL expired 2017/04/06 22:31:06.781436 [WARN] nomad.heartbeat: node 'f9f4c3de-f219-72f7-7e69-c40de400a536' TTL expired 2017/04/06 22:31:06.911462 [WARN] nomad.heartbeat: node '505afcd5-aba3-2f9e-bb90-f3524144970d' TTL expired 2017/04/06 22:31:06.927055 [WARN] nomad.heartbeat: node '6af7a69e-447c-a458-5a8c-d78e18fb6b63' TTL expired 2017/04/06 22:31:07.030944 [WARN] nomad.heartbeat: node '1d866133-8179-6939-19ff-7e265472c5ea' TTL expired 2017/04/06 22:31:07.132176 [WARN] nomad.heartbeat: node 'ce9aa9bb-fb34-8337-f7c9-a8f86fdc838e' TTL expired 2017/04/06 22:31:07.187500 [WARN] nomad.heartbeat: node '2189894c-a161-c2ba-02fc-97f4dda01f97' TTL expired 2017/04/06 22:31:07.504144 [WARN] nomad.heartbeat: node '6c1d0e7c-7245-c7f5-fdc8-1de8fbc34961' TTL expired 2017/04/06 22:31:07.656636 [WARN] nomad.heartbeat: node '5c97c8c9-924e-f5ec-7861-a818b429f542' TTL expired 2017/04/06 22:31:07.673002 [WARN] nomad.heartbeat: node 'a0a153e8-6670-abd5-ff69-1b845361dfd9' TTL expired 2017/04/06 22:31:07.754627 [WARN] nomad.heartbeat: node '117bbe8a-57fe-428a-94d0-362fae2c790b' TTL expired 2017/04/06 22:31:07.787169 [WARN] nomad.heartbeat: node '721dd5c2-371d-49d5-d6e7-6e2b4e64f0b6' TTL expired 2017/04/06 22:31:07.795460 [WARN] nomad.heartbeat: node '89bb1893-5ab6-2eaa-9c3d-69ff1c1bb676' TTL expired 2017/04/06 22:31:07.903492 [WARN] nomad.heartbeat: node 'e9db8ef8-7f80-b425-5345-64e75553f82b' TTL expired 2017/04/06 22:31:07.906233 [WARN] nomad.heartbeat: node '896ed93c-d263-6167-8cc8-1aab29558f17' TTL expired 2017/04/06 22:31:07.952019 [WARN] nomad.heartbeat: node '75324813-c002-739e-10e4-610e9315d148' TTL expired 2017/04/06 22:31:08.041792 [WARN] nomad.heartbeat: node '856c45fa-cafb-7fd8-f1c2-70e09226f722' TTL expired 2017/04/06 22:31:08.088045 [WARN] nomad.heartbeat: node 'd28e3099-9acb-1ea2-fbcf-5bb4283c606b' TTL expired 2017/04/06 22:31:08.098258 [WARN] nomad.heartbeat: node 'c08411ce-372b-81e2-1dd7-b4ecff3a81c0' TTL expired 2017/04/06 22:31:08.168311 [WARN] nomad.heartbeat: node '2df597af-3a1d-a565-ce0d-719953220bb7' TTL expired 2017/04/06 22:31:08.202649 [WARN] nomad.heartbeat: node '26723bd8-c95c-4327-4c16-1dc06f496815' TTL expired 2017/04/06 22:31:08.207617 [WARN] nomad.heartbeat: node 'cbf6ea10-3394-85f2-ca6e-97290c17b34c' TTL expired 2017/04/06 22:31:08.341941 [WARN] nomad.heartbeat: node '8f160ee2-257e-cdf4-75c6-82278eea7a9b' TTL expired 2017/04/06 22:31:08.372697 [WARN] nomad.heartbeat: node '38a88572-0fbf-69ed-cec9-96387a319a09' TTL expired 2017/04/06 22:31:08.377675 [WARN] nomad.heartbeat: node 'b62882f9-222f-31de-c220-dc6ee1942701' TTL expired 2017/04/06 22:31:08.429103 [WARN] nomad.heartbeat: node '6d07b01c-c611-fe28-abab-9f4c47606ada' TTL expired 2017/04/06 22:31:08.429376 [WARN] nomad.heartbeat: node '97fad29b-9839-82ab-3338-be47d245fdbe' TTL expired 2017/04/06 22:31:08.580201 [WARN] nomad.heartbeat: node 'c9c79fa1-35c8-b8ef-931a-03400fd2ad2e' TTL expired 2017/04/06 22:31:08.654641 [WARN] nomad.heartbeat: node '67a01676-8770-3b04-888d-b779f71533a1' TTL expired 2017/04/06 22:31:08.709772 [WARN] nomad.heartbeat: node '163a41cb-dd00-dfd4-94d5-874fccd2d057' TTL expired 2017/04/06 22:31:08.878011 [WARN] nomad.heartbeat: node 'cce94576-ca75-e5be-bf84-eb9e642627f3' TTL expired 2017/04/06 22:31:08.943219 [WARN] nomad.heartbeat: node '185a1a5f-bda8-1f83-a3f3-fd965df56259' TTL expired 2017/04/06 22:31:08.981687 [WARN] nomad.heartbeat: node '4c86115c-9e3d-f451-4da1-806158607f15' TTL expired 2017/04/06 22:31:08.993766 [WARN] nomad.heartbeat: node '12cd4099-c44d-2b82-c169-d4327dcb02cc' TTL expired 2017/04/06 22:31:09.086733 [WARN] nomad.heartbeat: node '21fbdf94-4ab4-423f-21c1-2bd55f8ff815' TTL expired 2017/04/06 22:31:09.138470 [WARN] nomad.heartbeat: node '4c83da3b-a7c1-c94d-a40e-a8b4c8dda101' TTL expired 2017/04/06 22:31:09.156312 [WARN] nomad.heartbeat: node 'd74db842-eb4b-f657-9b3d-2b1f4def86fc' TTL expired 2017/04/06 22:31:09.359110 [WARN] nomad.heartbeat: node '77336362-c028-8e8f-0842-881a930a9239' TTL expired 2017/04/06 22:31:09.363575 [WARN] nomad.heartbeat: node 'db4fb585-bdf6-a1d4-1a8f-9ab5df5031f7' TTL expired 2017/04/06 22:31:09.705670 [WARN] nomad.heartbeat: node 'e49145bb-d43f-b015-eec7-f0fc97ce39ac' TTL expired 2017/04/06 22:31:09.814382 [WARN] nomad.heartbeat: node '4da0644a-ee27-4fca-4476-5c3471c3e844' TTL expired 2017/04/06 22:31:09.829325 [WARN] nomad.heartbeat: node '6f8392c4-3d93-0c62-c96c-94cc37eb4265' TTL expired 2017/04/06 22:31:09.892986 [WARN] nomad.heartbeat: node '6359646f-25c2-4ad8-343f-92652ac58030' TTL expired 2017/04/06 22:31:09.908326 [WARN] nomad.heartbeat: node 'b95e1fc7-db83-4463-eba6-ce1ffa42ae77' TTL expired 2017/04/06 22:31:09.920482 [WARN] nomad.heartbeat: node 'f461808b-2836-d9c6-9d02-d6792c264434' TTL expired 2017/04/06 22:31:10.075771 [WARN] nomad.heartbeat: node 'ef0a88b8-7478-7efd-0802-239da822c922' TTL expired 2017/04/06 22:34:48 [INFO] raft: Restored from snapshot 3037-20413926-1491513528666 2017/04/06 22:34:48 [INFO] raft: Node at 10.201.0.8:4647 [Follower] entering Follower state (Leader: "") 2017/04/06 22:34:48 [INFO] serf: EventMemberJoin: ip-10-201-0-8.us-west 10.201.0.8 2017/04/06 22:34:48.338595 [INFO] nomad: starting 4 scheduling worker(s) for [batch system service _core] 2017/04/06 22:34:48.338953 [INFO] nomad: adding server ip-10-201-0-8.us-west (Addr: 10.201.0.8:4647) (DC: aws-us-west-1) 2017/04/06 22:34:48 [INFO] serf: Attempting re-join to previously known node: ip-10-201-4-6.us-west: 10.201.4.6:4648 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48.341108 [INFO] agent: Joining cluster... 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [INFO] serf: EventMemberJoin: ip-10-201-4-6.us-west 10.201.4.6 2017/04/06 22:34:48 [INFO] serf: EventMemberJoin: ip-10-201-0-6.us-west 10.201.0.6 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [INFO] serf: EventMemberJoin: ip-10-201-5-6.us-west 10.201.5.6 2017/04/06 22:34:48 [INFO] serf: EventMemberJoin: ip-10-201-0-7.us-west 10.201.0.7 2017/04/06 22:34:48 [INFO] serf: Re-joined to previously known node: ip-10-201-4-6.us-west: 10.201.4.6:4648 2017/04/06 22:34:48.348769 [INFO] nomad: adding server ip-10-201-4-6.us-west (Addr: 10.201.4.6:4647) (DC: aws-us-west-1) 2017/04/06 22:34:48.348787 [INFO] nomad: adding server ip-10-201-0-6.us-west (Addr: 10.201.0.6:4647) (DC: aws-us-west-1) 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48.348868 [INFO] nomad: adding server ip-10-201-5-6.us-west (Addr: 10.201.5.6:4647) (DC: aws-us-west-1) 2017/04/06 22:34:48.348882 [INFO] nomad: adding server ip-10-201-0-7.us-west (Addr: 10.201.0.7:4647) (DC: aws-us-west-1) 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [WARN] yamux: failed to send ping reply: session shutdown 2017/04/06 22:34:48 [ERR] yamux: Failed to write header: write tcp 10.201.0.8:4647->10.201.4.179:59508: use of closed network connection 2017/04/06 22:34:48.352358 [ERR] nomad.rpc: multiplex conn accept failed: session shutdown 2017/04/06 22:34:48 [ERR] yamux: Failed to write header: write tcp 10.201.0.8:4647->10.201.2.124:41158: use of closed network connection 2017/04/06 22:34:48 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 22:34:48.378553 [INFO] agent: Join completed. Synced with 5 initial agents 2017/04/06 22:34:48.390112 [INFO] server.consul: successfully contacted 4 Nomad Servers 2017/04/06 22:34:48 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 22:34:49 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 22:34:49 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:50 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:50 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:51 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:51 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:51 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:51 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:51 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:51 [WARN] raft: Rejecting vote request from 10.201.0.6:4647 since our last term is greater (3042, 3039) 2017/04/06 22:34:51 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:51 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:51 [WARN] raft: Clearing log suffix from 20417522 to 20417585 2017/04/06 22:34:51 [WARN] raft: Skipping application of old log: 20417585 2017/04/06 22:34:53 [WARN] raft: Heartbeat timeout from "10.201.0.6:4647" reached, starting election 2017/04/06 22:34:53 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 22:34:55 [WARN] raft: Election timeout reached, restarting election 2017/04/06 22:34:55 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 22:34:57 [WARN] raft: Election timeout reached, restarting election 2017/04/06 22:34:57 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 22:34:58 [WARN] raft: Election timeout reached, restarting election 2017/04/06 22:34:58 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 22:34:58 [INFO] raft: Node at 10.201.0.8:4647 [Follower] entering Follower state (Leader: "") 2017/04/06 22:34:58 [WARN] raft: Failed to get previous log: 20418403 log not found (last: 20417585) 2017/04/06 22:38:10 [ERR] nomad.fsm: DeleteJob failed: job not found 2017/04/06 22:40:05.353180 [ERR] worker: failed to submit plan for evaluation 9aa933c6-717d-69a7-1781-620417ebc206: rpc error: EOF 2017/04/06 22:40:05.353456 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 22:40:05.353469 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 22:40:05.353631 [ERR] worker: failed to dequeue evaluation: rpc error: EOF 2017/04/06 22:40:05.357551 [ERR] worker: failed to nack evaluation '9aa933c6-717d-69a7-1781-620417ebc206': rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:05.359031 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.373933 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.373972 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.374203 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:05.379575 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:05.454543 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:05.454590 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.454717 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.460097 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:05 [WARN] raft: Rejecting vote request from 10.201.4.6:4647 since we have a leader: 10.201.0.6:4647 2017/04/06 22:40:05.775143 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:05.775203 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.775233 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: rpc error: lead thread didn't get connection 2017/04/06 22:40:05.780670 [ERR] worker: failed to dequeue evaluation: rpc error: failed to get conn: dial tcp 10.201.0.6:4647: getsockopt: connection refused 2017/04/06 22:40:06 [WARN] raft: Rejecting vote request from 10.201.0.7:4647 since we have a leader: 10.201.0.6:4647 2017/04/06 22:40:06 [WARN] raft: Rejecting vote request from 10.201.5.6:4647 since we have a leader: 10.201.0.6:4647 2017/04/06 22:40:07 [WARN] raft: Heartbeat timeout from "10.201.0.6:4647" reached, starting election 2017/04/06 22:40:07 [INFO] raft: Node at 10.201.0.8:4647 [Candidate] entering Candidate state 2017/04/06 22:40:07 [ERR] raft: Failed to make RequestVote RPC to 10.201.0.6:4647: EOF 2017/04/06 22:40:07 [INFO] raft: Node at 10.201.0.8:4647 [Follower] entering Follower state (Leader: "") 2017/04/06 22:40:33 [INFO] memberlist: Suspect ip-10-201-0-6.us-west has failed, no acks received 2017/04/06 22:40:46 [ERR] yamux: keepalive failed: i/o deadline reached 2017/04/06 22:40:46.718888 [ERR] http: Request /v1/allocation/f41f8a5c-d753-035b-dc89-c44c93eb4308?index=20418557&wait=30s, error: rpc error: EOF 2017/04/06 22:40:46.718889 [ERR] http: Request /v1/allocation/f589ab80-357f-a11a-f48c-09587a9d1696?index=20418557&wait=30s, error: rpc error: EOF 2017/04/06 22:40:46.719020 [ERR] http: Request /v1/allocation/feda2846-c2e8-406d-4804-30ac787dfbe5?index=20418557&wait=30s, error: rpc error: EOF 2017/04/06 22:40:54 [INFO] serf: EventMemberFailed: ip-10-201-0-6.us-west 10.201.0.6 2017/04/06 22:40:54.747177 [INFO] nomad: removing server ip-10-201-0-6.us-west (Addr: 10.201.0.6:4647) (DC: aws-us-west-1) 2017/04/06 22:41:08 [INFO] memberlist: Suspect ip-10-201-0-6.us-west has failed, no acks received 2017/04/06 22:42:48 [INFO] serf: attempting reconnect to ip-10-201-0-6.us-west 10.201.0.6:4648 2017/04/06 22:43:48 [INFO] serf: EventMemberJoin: ip-10-201-0-6.us-west 10.201.0.6 2017/04/06 22:43:48.502980 [INFO] nomad: adding server ip-10-201-0-6.us-west (Addr: 10.201.0.6:4647) (DC: aws-us-west-1) 2017/04/06 22:43:51 [WARN] raft: Rejecting vote request from 10.201.0.6:4647 since we have a leader: 10.201.4.6:4647 2017/04/06 22:43:52.949545 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled 2017/04/06 22:43:52.949623 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled 2017/04/06 22:43:52.949706 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled 2017/04/06 22:43:52.949805 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled 2017/04/06 22:43:53 [WARN] raft: Rejecting vote request from 10.201.0.6:4647 since we have a leader: 10.201.4.6:4647 2017/04/06 22:43:54.638151 [ERR] worker: failed to dequeue evaluation: rpc error: eval broker disabled