Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nomad 0.9 - Turn-off VM sometimes cause not starting our ruby webserver #5613

Closed
jozef-slezak opened this issue Apr 25, 2019 · 8 comments
Closed

Nomad 0.9 - Turn-off VM sometimes cause not starting our ruby webserver #5613

jozef-slezak opened this issue Apr 25, 2019 · 8 comments
Assignees
@notnoop
Labels
stage/needs-investigation theme/client type/bug

Comments

@jozef-slezak
Copy link

jozef-slezak commented Apr 25, 2019 •
edited

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Nomad v0.9.0-rc2 (6a838b8)
Nomad v0.9.1-rc1 (3e17899)

Operating system and Environment details

Linux innoabis-71 3.10.0-957.10.1.el7.x86_64 #1 SMP Mon Mar 18 15:06:45 UTC 2019 x86_64 x86_64 x86_64 GNU/Linux

Issue

Reproduction steps

Virtualbox VM
Start Nomad with some raw_exec processes
Turn-off VM
Start VM
(Repeat until at least one process remains DEAD even after few minues)

You can see the error in Nomad UI
http://localhost:4646/ui/allocations/16f7e5e1-5f3c-086e-906c-54c994707dfb/WebAFIS-Webserver

Apr 24, '19 18:34:19 +0200 Killed Task successfully killed
Apr 24, '19 18:34:18 +0200 Terminated Exit Code: 0, Exit Message: "executor: error waiting on process: rpc error: code = Canceled desc = grpc: the client connection is closing"
Apr 24, '19 18:34:18 +0200 Killing Sent interrupt
Apr 24, '19 15:18:24 +0200 Started Task started by client
Apr 24, '19 15:18:23 +0200 Task Setup Building Task Directory
Apr 24, '19 15:18:23 +0200 Received Task received by client

I am able to start the job from the web ui manualy (my problem is that it havent started automatically).
It seems that the job is not for some reason scheduled (it stays "Task successfully killed" but it should be started). Strange because no other job is doing that (could that cause the shell script or timeouts timeouts?).

Tried several more times (sometimes new VM turn-off helped and everything started). Now I see in nomad web console "Nomad will attempt to reschedule an hour ago" DEAD "Nomad will attempt to reschedule an hour ago". Isn't it just about synchronization between nomad internal processes?

Apr 25, '19 15:32:43 +0200 Not Restarting Error was unrecoverable
Apr 25, '19 15:32:43 +0200 Task hook failed logmon: rpc error: code = Unknown desc = failed to create stdout logfile for "ABIS-Monitoring.stdout": open /var/lib/nomad/alloc/26a022e0-9075-fc3a-5571-93940ece0460/alloc/logs: no such file or directory
Apr 25, '19 15:32:28 +0200 Restarting Task restarting in 15.193920007s
Apr 25, '19 15:32:28 +0200 Terminated Exit Code: 0
Apr 25, '19 15:32:07 +0200 Started Task started by client
Apr 25, '19 15:32:04 +0200 Task Setup Building Task Directory
Apr 25, '19 15:32:04 +0200 Received Task received by client
Apr 25, '19 15:31:58 +0200 Started Task started by client
Apr 25, '19 15:20:01 +0200 Not Restarting Exceeded allowed attempts 2 in interval 30m0s and mode is "fail"
Apr 25, '19 15:20:01 +0200 Terminated Exit Code: 0

Job file (if appropriate)

raw exec

...
"Config": {
            "args": [
              "-c",
              "cd /opt/webserver && exec /bin/bash ./bin/production_run_webserver.sh"
            ],
            "command": "/bin/bash"
          },
...

Nomad Client logs (if appropriate)

If possible please post relevant logs in the issue.

Logs and other artifacts may also be sent to: nomad-oss-debug@hashicorp.com

Please link to your Github issue in the email and reference it in the subject
line:

journalctl -u nomad
-- Logs begin at Wed 2019-04-24 14:51:15 CEST, end at Wed 2019-04-24 22:39:46 CEST. --
Apr 24 14:51:37 innoabis-71 systemd[1]: Started HashiCorp Nomad.
Apr 24 14:51:38 innoabis-71 nomad[3322]: WARNING: keyring exists but -encrypt given, using keyring
Apr 24 14:51:38 innoabis-71 nomad[3322]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Apr 24 14:51:38 innoabis-71 nomad[3322]: ==> Loaded configuration from /etc/nomad.d/nomad-config.hcl, /etc/nomad.d/nomad.hcl
Apr 24 14:51:38 innoabis-71 nomad[3322]: ==> Starting Nomad agent...
Apr 24 14:51:48 innoabis-71 nomad[3322]: ==> Nomad agent configuration:
Apr 24 14:51:48 innoabis-71 nomad[3322]: Advertise Addrs: HTTP: 10.0.2.15:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648
Apr 24 14:51:48 innoabis-71 nomad[3322]: Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648
Apr 24 14:51:48 innoabis-71 nomad[3322]: Client: true
Apr 24 14:51:48 innoabis-71 nomad[3322]: Log Level: INFO
Apr 24 14:51:48 innoabis-71 nomad[3322]: Region: global (DC: dc1)
Apr 24 14:51:48 innoabis-71 nomad[3322]: Server: true
Apr 24 14:51:48 innoabis-71 nomad[3322]: Version: 0.9.0-rc2
Apr 24 14:51:48 innoabis-71 nomad[3322]: ==> Nomad agent started! Log data will stream in below:
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.887+0200 [WARN ] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/plugins
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:38.940+0200 [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.112+0200 [INFO ] nomad: raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.0.2.15:4647 Address:10.0.2.15:4647}]
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.113+0200 [INFO ] nomad: raft: Node at 10.0.2.15:4647 [Follower] entering Follower state (Leader: "")
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.163+0200 [INFO ] nomad: serf: EventMemberJoin: innoabis-71.global 10.0.2.15
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.164+0200 [INFO ] nomad: starting scheduling worker(s): num_workers=2 schedulers="[service batch system _core]"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.174+0200 [INFO ] nomad: adding server: server="innoabis-71.global (Addr: 10.0.2.15:4647) (DC: dc1)"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.174+0200 [WARN ] nomad: serf: Failed to re-join any previously known node
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.187+0200 [INFO ] client: using state directory: state_dir=/var/lib/nomad/client
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.207+0200 [INFO ] client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.238+0200 [INFO ] client.fingerprint_mgr.cgroup: cgroups are available
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.255+0200 [INFO ] client.fingerprint_mgr.consul: consul agent is available
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.284+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=driver
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.284+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=device
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.734+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=2232a445-10cc-3ae6-3056-92a9016f3
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.734+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 error="failed to reattach to executor: error creating rpc client for ex
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.760+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=2d7547bd-e955-70bf-587e-fc5af971f
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.792+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=67b0a2cd-2072-9a0c-8068-f3661226b
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.792+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service error="failed to reattach to executor: error creating rpc client for e
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.808+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=996081a7-5642-6ad8-2e73-057b3e737
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.808+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403 task=ABIS-Monitoring error="failed to reattach to executor: error creating rpc client fo
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.847+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=bd88dbd7-a073-a078-2677-48f5fc0ac
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.847+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode error="failed to reattach to executor: error creating rpc client for executor
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.858+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=d0a12c58-e706-ad48-191f-6bbf40ecf
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.874+0200 [INFO ] client: started client: node_id=d2ff5fcf-7040-71cb-89d5-25c8a1ba4f35
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.882+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to launch logmon process: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode error="Reattachment process not found"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.890+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to launch logmon process: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 error="Reattachment process not found"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.905+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to launch logmon process: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 error="Reattachment process not found"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.915+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to launch logmon process: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service error="Reattachment process not found"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.925+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to launch logmon process: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403 task=ABIS-Monitoring error="Reattachment process not found"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=762568ec-0ca1-dd25-0f5b-84b404aa672b
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=03cf15be-fbd0-72dc-0a41-688ed3a5c96f
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=337e2138-5084-635a-2752-d119c7640b2a
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=58b7ebbb-9761-7a2c-d434-f163a7dbedde
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f64e09aa-7425-70b4-e770-c158c03285fd
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=41aa95e6-5f8d-b69b-9576-bcc1752e0a3a
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=a19db4cc-e6ec-5cdf-a2c6-912d524b253d
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=b50e56fa-91e1-8e3c-754d-49802c364a93
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=64978202-6ca0-df9b-7af8-4f3ff010b47d
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=ffd88753-3f61-3fad-4652-c6347a4678e7
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=0839351f-0605-1086-db26-a65779e5b111
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=052e2a63-d1dd-293d-9265-ccf70af45498
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f70f1336-e523-95a1-88ae-d1b2f2e09734
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=866c09df-221c-0f94-db69-58e8e4b535ee
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.927+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2748dd53-e986-dbd1-0e20-ed21c847c418
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=762568ec-0ca1-dd25-0f5b-84b404aa672b reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=03cf15be-fbd0-72dc-0a41-688ed3a5c96f reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=337e2138-5084-635a-2752-d119c7640b2a reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=58b7ebbb-9761-7a2c-d434-f163a7dbedde reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f64e09aa-7425-70b4-e770-c158c03285fd reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=41aa95e6-5f8d-b69b-9576-bcc1752e0a3a reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=a19db4cc-e6ec-5cdf-a2c6-912d524b253d reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=b50e56fa-91e1-8e3c-754d-49802c364a93 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=64978202-6ca0-df9b-7af8-4f3ff010b47d reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=ffd88753-3f61-3fad-4652-c6347a4678e7 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=0839351f-0605-1086-db26-a65779e5b111 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=052e2a63-d1dd-293d-9265-ccf70af45498 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f70f1336-e523-95a1-88ae-d1b2f2e09734 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=866c09df-221c-0f94-db69-58e8e4b535ee reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.928+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2748dd53-e986-dbd1-0e20-ed21c847c418 reason="forced collection"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.983+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode error="prestart hook "logmon" failed: Reattachment process not found"
Apr 24 14:51:48 innoabis-71 nomad[3322]: 2019-04-24T14:51:48.983+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode reason="Restart within policy" delay=15.242787453s
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.318+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 error="prestart hook "logmon" failed: Reattachment process not found"
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.318+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 reason="Restart within policy" delay=15.242787453s
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.504+0200 [WARN ] nomad: raft: Heartbeat timeout from "" reached, starting election
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.504+0200 [INFO ] nomad: raft: Node at 10.0.2.15:4647 [Candidate] entering Candidate state in term 14
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.605+0200 [INFO ] nomad: raft: Election won. Tally: 1
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.605+0200 [INFO ] nomad: raft: Node at 10.0.2.15:4647 [Leader] entering Leader state
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.608+0200 [INFO ] nomad: cluster leadership acquired
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.618+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 error="prestart hook "logmon" failed: Reattachment process not found"
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.618+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 reason="Restart within policy" delay=15.242787453s
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.740+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service error="prestart hook "logmon" failed: Reattachment process not found"
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.740+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service reason="Restart within policy" delay=15.242787453s
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.756+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403 task=ABIS-Monitoring error="prestart hook "logmon" failed: Reattachment process not found"
Apr 24 14:51:49 innoabis-71 nomad[3322]: 2019-04-24T14:51:49.756+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403 task=ABIS-Monitoring reason="Restart within policy" delay=15.242787453s
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.118+0200 [INFO ] client: node registration complete
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.173+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/e0d67b3c-1a
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.173+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/e0d6
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.174+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/e0d67b3c-1a
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.174+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/e0d6
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.318+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.318+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.332+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/e0d67b3c-
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.333+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/e0d67b3c-
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.377+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.377+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.378+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor path=/var/lib/nomad/alloc/f30a8688-ef0d-6344-a8f2-a7
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.379+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/f30a8688-ef
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.535+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/d0a12c58-e706
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.535+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring path=/var/lib/nomad/alloc/d0a12c58-e706-ad48-191f-6bbf
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.695+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=337e2138-5084-635a-2752-d119c7640b2a
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.795+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to start logmon: alloc_id=337e2138-5084-635a-2752-d119c7640b2a task=Fast-Service error="rpc error: code = Unknown desc = failed to create stdout logfile
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.854+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=b50e56fa-91e1-8e3c-754d-49802c364a93
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.877+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 @module=logmon path=/var/lib/nomad/alloc/2d7547bd-e955-70b
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.877+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 @module=logmon path=/var/lib/nomad/alloc/2d7547bd-e955-70b
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.926+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=337e2138-5084-635a-2752-d119c7640b2a task=Fast-Service error="prestart hook "logmon" failed: rpc error: code = Unknown desc = failed to create s
Apr 24 14:51:50 innoabis-71 nomad[3322]: 2019-04-24T14:51:50.926+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=337e2138-5084-635a-2752-d119c7640b2a task=Fast-Service reason="Error was unrecoverable"
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.091+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[/opt/abis-monitoring/bin/abis-monitoring.sh]}"
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.126+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=03cf15be-fbd0-72dc-0a41-688ed3a5c96f
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.180+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.263+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=866c09df-221c-0f94-db69-58e8e4b535ee
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.270+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.323+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=58b7ebbb-9761-7a2c-d434-f163a7dbedde
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.356+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=41aa95e6-5f8d-b69b-9576-bcc1752e0a3a
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.357+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=a19db4cc-e6ec-5cdf-a2c6-912d524b253d
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.411+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=052e2a63-d1dd-293d-9265-ccf70af45498
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.445+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=0839351f-0605-1086-db26-a65779e5b111
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.483+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.483+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.530+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f70f1336-e523-95a1-88ae-d1b2f2e09734
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.548+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2748dd53-e986-dbd1-0e20-ed21c847c418
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.707+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=64978202-6ca0-df9b-7af8-4f3ff010b47d
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.736+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=762568ec-0ca1-dd25-0f5b-84b404aa672b
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.736+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=ffd88753-3f61-3fad-4652-c6347a4678e7
Apr 24 14:51:51 innoabis-71 nomad[3322]: 2019-04-24T14:51:51.857+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.027+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f64e09aa-7425-70b4-e770-c158c03285fd
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.037+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/f30a8688-ef
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.037+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/f30a8688-ef
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.210+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 @module=logmon path=/var/lib/nomad/alloc/2232a445-10cc-3ae
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.210+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 path=/var/lib/nomad/alloc/2232a445-10cc-3ae6-3056-92a9016f
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.399+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/67b0a2cd-2072-9a
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.399+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/67b0a2cd-2072-9a
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.446+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /srv/apps/eafis/eafis-cluster/eafis.sh]}"
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.487+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode @module=logmon path=/var/lib/nomad/alloc/bd88dbd7-a073-a078-2677
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.487+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode path=/var/lib/nomad/alloc/bd88dbd7-a073-a078-2677-48f5fc0ac575/a
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.662+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.876+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 31849]}"
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.933+0200 [WARN ] nomad: failed to query Nomad service in Consul datacenter: service_name=nomad dc=dc1 error="failed to query service "nomad" in Consul datacenter "dc1": Get http://127.0.0.1:8500/v1/catalog/service/nomad?
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.933+0200 [ERROR] nomad: error looking up Nomad servers in Consul: error="1 error(s) occurred:
Apr 24 14:51:52 innoabis-71 nomad[3322]: * failed to query service "nomad" in Consul datacenter "dc1": Get http://127.0.0.1:8500/v1/catalog/service/nomad?dc=dc1&near=_agent&stale=&tag=serf&wait=2000ms: net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Apr 24 14:51:52 innoabis-71 nomad[3322]: 2019-04-24T14:51:52.959+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./INode -c inode-config.xml -d 1]}"
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.070+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b driver=raw_exec task_name=activemq515 error="rpc error: code = Un
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.078+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 reason="Restart within policy" delay=18.379557727s
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.083+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: failed to read from log fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor @module=logmon error="read /var/lib/
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.083+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: failed to read from log fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor @module=logmon error="read /var/lib/
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.091+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.253+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.556+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403
Apr 24 14:51:53 innoabis-71 nomad[3322]: 2019-04-24T14:51:53.829+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575
Apr 24 14:51:54 innoabis-71 nomad[3322]: 2019-04-24T14:51:54.294+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680
Apr 24 14:51:54 innoabis-71 nomad[3322]: 2019-04-24T14:51:54.489+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b
Apr 24 14:51:54 innoabis-71 nomad[3322]: 2019-04-24T14:51:54.492+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service error="read |0: file already closed"
Apr 24 14:51:54 innoabis-71 nomad[3322]: 2019-04-24T14:51:54.612+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.153+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server @modul
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.153+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server @modul
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.153+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker @module=
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.156+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver @module=logmo
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.156+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver @module=logmo
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.160+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver error="read |0: file already closed"
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.170+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker @module=
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.170+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker @module=
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.171+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker @module=
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.175+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker error="read |0: file already closed"
Apr 24 14:51:55 innoabis-71 nomad[3322]: 2019-04-24T14:51:55.176+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker error="read |0: file already closed"
Apr 24 14:51:56 innoabis-71 nomad[3322]: 2019-04-24T14:51:56.237+0200 [INFO ] client: node registration complete
Apr 24 14:52:09 innoabis-71 nomad[3322]: 2019-04-24T14:52:09.395+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 driver=raw_exec task_name=activemq515 error="rpc error: code = Un
Apr 24 14:52:09 innoabis-71 nomad[3322]: 2019-04-24T14:52:09.396+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 reason="Restart within policy" delay=17.570943498s
Apr 24 14:52:11 innoabis-71 nomad[3322]: 2019-04-24T14:52:11.532+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to start logmon: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 error="rpc error: code = Unknown desc = failed to create stdout logfile f
Apr 24 14:52:11 innoabis-71 nomad[3322]: 2019-04-24T14:52:11.538+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 error="prestart hook "logmon" failed: rpc error: code = Unknown desc = failed to create st
Apr 24 14:52:11 innoabis-71 nomad[3322]: 2019-04-24T14:52:11.538+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 reason="Error was unrecoverable"
Apr 24 14:52:27 innoabis-71 nomad[3322]: 2019-04-24T14:52:27.001+0200 [ERROR] client.alloc_runner.task_runner.task_hook: failed to start logmon: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 error="rpc error: code = Unknown desc = failed to create stdout logfile f
Apr 24 14:52:27 innoabis-71 nomad[3322]: 2019-04-24T14:52:27.015+0200 [ERROR] client.alloc_runner.task_runner: prestart failed: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 error="prestart hook "logmon" failed: rpc error: code = Unknown desc = failed to create st
Apr 24 14:52:27 innoabis-71 nomad[3322]: 2019-04-24T14:52:27.015+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 reason="Error was unrecoverable"
Apr 24 14:52:27 innoabis-71 nomad[3322]: 2019-04-24T14:52:27.046+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 error="read |0: file already closed"
Apr 24 14:52:41 innoabis-71 nomad[3322]: 2019-04-24T14:52:41.749+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=abc490f7-b0d5-f4f0-d673-d73489c029f9 task=activemq515 path=/var/lib/nomad/alloc/abc490f7-b0d5-f4f0-d673-d73489c0
Apr 24 14:52:41 innoabis-71 nomad[3322]: 2019-04-24T14:52:41.749+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=abc490f7-b0d5-f4f0-d673-d73489c029f9 task=activemq515 @module=logmon path=/var/lib/nomad/alloc/abc490f7-b0d5-f4f
Apr 24 14:52:41 innoabis-71 nomad[3322]: 2019-04-24T14:52:41.796+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 24 14:52:42 innoabis-71 nomad[3322]: 2019-04-24T14:52:42.839+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:29 innoabis-71 nomad[3322]: 2019-04-24T14:53:29.762+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:30 innoabis-71 nomad[3322]: 2019-04-24T14:53:30.937+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:41 innoabis-71 nomad[3322]: 2019-04-24T14:53:41.106+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:43 innoabis-71 nomad[3322]: 2019-04-24T14:53:43.270+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:44 innoabis-71 nomad[3322]: 2019-04-24T14:53:44.385+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:45 innoabis-71 nomad[3322]: 2019-04-24T14:53:45.285+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:53:46 innoabis-71 nomad[3322]: 2019-04-24T14:53:46.136+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:05 innoabis-71 nomad[3322]: 2019-04-24T14:54:05.908+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:14 innoabis-71 nomad[3322]: 2019-04-24T14:54:14.067+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:15 innoabis-71 nomad[3322]: 2019-04-24T14:54:15.500+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:16 innoabis-71 nomad[3322]: 2019-04-24T14:54:16.430+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:17 innoabis-71 nomad[3322]: 2019-04-24T14:54:17.277+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:18 innoabis-71 nomad[3322]: 2019-04-24T14:54:18.264+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:19 innoabis-71 nomad[3322]: 2019-04-24T14:54:19.129+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:20 innoabis-71 nomad[3322]: 2019-04-24T14:54:20.268+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:31 innoabis-71 nomad[3322]: 2019-04-24T14:54:31.096+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:39 innoabis-71 nomad[3322]: 2019-04-24T14:54:39.436+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:40 innoabis-71 nomad[3322]: 2019-04-24T14:54:40.665+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:41 innoabis-71 nomad[3322]: 2019-04-24T14:54:41.729+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:42 innoabis-71 nomad[3322]: 2019-04-24T14:54:42.869+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:43 innoabis-71 nomad[3322]: 2019-04-24T14:54:43.768+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:44 innoabis-71 nomad[3322]: 2019-04-24T14:54:44.702+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:45 innoabis-71 nomad[3322]: 2019-04-24T14:54:45.546+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:46 innoabis-71 nomad[3322]: 2019-04-24T14:54:46.470+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:47 innoabis-71 nomad[3322]: 2019-04-24T14:54:47.436+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:54:48 innoabis-71 nomad[3322]: 2019-04-24T14:54:48.342+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:23 innoabis-71 nomad[3322]: 2019-04-24T14:55:23.070+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:24 innoabis-71 nomad[3322]: 2019-04-24T14:55:24.257+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:25 innoabis-71 nomad[3322]: 2019-04-24T14:55:25.222+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:34 innoabis-71 nomad[3322]: 2019-04-24T14:55:34.343+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:35 innoabis-71 nomad[3322]: 2019-04-24T14:55:35.833+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:54 innoabis-71 nomad[3322]: 2019-04-24T14:55:54.246+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:55 innoabis-71 nomad[3322]: 2019-04-24T14:55:55.513+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:56 innoabis-71 nomad[3322]: 2019-04-24T14:55:56.462+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:57 innoabis-71 nomad[3322]: 2019-04-24T14:55:57.399+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:58 innoabis-71 nomad[3322]: 2019-04-24T14:55:58.450+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:55:59 innoabis-71 nomad[3322]: 2019-04-24T14:55:59.981+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:00 innoabis-71 nomad[3322]: 2019-04-24T14:56:00.766+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:01 innoabis-71 nomad[3322]: 2019-04-24T14:56:01.739+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:02 innoabis-71 nomad[3322]: 2019-04-24T14:56:02.746+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:07 innoabis-71 nomad[3322]: 2019-04-24T14:56:07.715+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:08 innoabis-71 nomad[3322]: 2019-04-24T14:56:08.975+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:25 innoabis-71 nomad[3322]: 2019-04-24T14:56:25.630+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:38 innoabis-71 nomad[3322]: 2019-04-24T14:56:38.933+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:40 innoabis-71 nomad[3322]: 2019-04-24T14:56:40.067+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:41 innoabis-71 nomad[3322]: 2019-04-24T14:56:41.284+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:56:48 innoabis-71 nomad[3322]: 2019-04-24T14:56:48.890+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:57:28 innoabis-71 nomad[3322]: 2019-04-24T14:57:28.427+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:57:36 innoabis-71 nomad[3322]: 2019-04-24T14:57:36.688+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:57:38 innoabis-71 nomad[3322]: 2019-04-24T14:57:38.776+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 14:57:46 innoabis-71 nomad[3322]: 2019-04-24T14:57:46.874+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 14:57:48 innoabis-71 nomad[3322]: 2019-04-24T14:57:48.468+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stderr error="failed to list entries: open /var/li
Apr 24 14:58:07 innoabis-71 nomad[3322]: 2019-04-24T14:58:07.117+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/d0a12c58-e706-ad48-191f-6bbf40ecf631?follow=true&offset=50000&origin=end&task=ABIS-Monitoring&type=stdout error="failed to list entries: open /var
Apr 24 14:58:09 innoabis-71 nomad[3322]: 2019-04-24T14:58:09.071+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/d0a12c58-e706-ad48-191f-6bbf40ecf631?follow=true&offset=50000&origin=end&task=ABIS-Monitoring&type=stderr error="failed to list entries: open /var
Apr 24 14:58:09 innoabis-71 nomad[3322]: 2019-04-24T14:58:09.700+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/d0a12c58-e706-ad48-191f-6bbf40ecf631?follow=true&offset=50000&origin=end&task=ABIS-Monitoring&type=stdout error="failed to list entries: open /var
Apr 24 14:58:11 innoabis-71 nomad[3322]: 2019-04-24T14:58:11.103+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/d0a12c58-e706-ad48-191f-6bbf40ecf631?follow=true&offset=50000&origin=end&task=ABIS-Monitoring&type=stderr error="failed to list entries: open /var
Apr 24 14:58:11 innoabis-71 nomad[3322]: 2019-04-24T14:58:11.494+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/d0a12c58-e706-ad48-191f-6bbf40ecf631?follow=true&offset=50000&origin=end&task=ABIS-Monitoring&type=stdout error="failed to list entries: open /var
Apr 24 14:58:18 innoabis-71 nomad[3322]: 2019-04-24T14:58:18.711+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:19 innoabis-71 nomad[3322]: 2019-04-24T14:58:19.805+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:20 innoabis-71 nomad[3322]: 2019-04-24T14:58:20.694+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:21 innoabis-71 nomad[3322]: 2019-04-24T14:58:21.738+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:34 innoabis-71 nomad[3322]: 2019-04-24T14:58:34.679+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:36 innoabis-71 nomad[3322]: 2019-04-24T14:58:36.073+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:37 innoabis-71 nomad[3322]: 2019-04-24T14:58:37.564+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:57 innoabis-71 nomad[3322]: 2019-04-24T14:58:57.644+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:58:59 innoabis-71 nomad[3322]: 2019-04-24T14:58:59.569+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:00 innoabis-71 nomad[3322]: 2019-04-24T14:59:00.615+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:05 innoabis-71 nomad[3322]: 2019-04-24T14:59:05.101+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:06 innoabis-71 nomad[3322]: 2019-04-24T14:59:06.552+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:07 innoabis-71 nomad[3322]: 2019-04-24T14:59:07.950+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:09 innoabis-71 nomad[3322]: 2019-04-24T14:59:09.167+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:10 innoabis-71 nomad[3322]: 2019-04-24T14:59:10.436+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 14:59:11 innoabis-71 nomad[3322]: 2019-04-24T14:59:11.362+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:01:12 innoabis-71 nomad[3322]: 2019-04-24T15:01:12.787+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:02:04 innoabis-71 nomad[3322]: 2019-04-24T15:02:04.070+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:02:06 innoabis-71 nomad[3322]: 2019-04-24T15:02:06.969+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:02:08 innoabis-71 nomad[3322]: 2019-04-24T15:02:08.448+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:03:30 innoabis-71 nomad[3322]: 2019-04-24T15:03:30.176+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/75e56125-41cf-284
Apr 24 15:03:30 innoabis-71 nomad[3322]: 2019-04-24T15:03:30.176+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher path=/var/lib/nomad/alloc/75e56125-41cf-284d-a99e-3e99342d
Apr 24 15:03:30 innoabis-71 nomad[3322]: 2019-04-24T15:03:30.200+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 15:03:32 innoabis-71 nomad[3322]: 2019-04-24T15:03:32.453+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 15:03:32 innoabis-71 nomad[3322]: 2019-04-24T15:03:32.453+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 15:03:32 innoabis-71 nomad[3322]: 2019-04-24T15:03:32.453+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher reason="Restart within policy" delay=16.628734632s
Apr 24 15:03:37 innoabis-71 nomad[3322]: 2019-04-24T15:03:37.301+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:03:49 innoabis-71 nomad[3322]: 2019-04-24T15:03:49.096+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/75e56125-41cf-284
Apr 24 15:03:49 innoabis-71 nomad[3322]: 2019-04-24T15:03:49.097+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/75e56125-41cf-284
Apr 24 15:03:49 innoabis-71 nomad[3322]: 2019-04-24T15:03:49.098+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 15:03:51 innoabis-71 nomad[3322]: 2019-04-24T15:03:51.227+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 15:03:51 innoabis-71 nomad[3322]: 2019-04-24T15:03:51.230+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher reason="Restart within policy" delay=18.057529016s
Apr 24 15:04:09 innoabis-71 nomad[3322]: 2019-04-24T15:04:09.317+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher path=/var/lib/nomad/alloc/75e56125-41cf-284d-a99e-3e99342d
Apr 24 15:04:09 innoabis-71 nomad[3322]: 2019-04-24T15:04:09.317+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/75e56125-41cf-284
Apr 24 15:04:09 innoabis-71 nomad[3322]: 2019-04-24T15:04:09.319+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 15:04:11 innoabis-71 nomad[3322]: 2019-04-24T15:04:11.455+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 15:04:11 innoabis-71 nomad[3322]: 2019-04-24T15:04:11.457+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 15:04:11 innoabis-71 nomad[3322]: 2019-04-24T15:04:11.485+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae
Apr 24 15:08:14 innoabis-71 nomad[3322]: 2019-04-24T15:08:14.453+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:16 innoabis-71 nomad[3322]: 2019-04-24T15:08:16.093+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:17 innoabis-71 nomad[3322]: 2019-04-24T15:08:17.315+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:18 innoabis-71 nomad[3322]: 2019-04-24T15:08:18.268+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:21 innoabis-71 nomad[3322]: 2019-04-24T15:08:21.953+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:29 innoabis-71 nomad[3322]: 2019-04-24T15:08:29.126+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 15:08:29 innoabis-71 nomad[3322]: 2019-04-24T15:08:29.368+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stderr error="failed to list entries: open /var/li
Apr 24 15:08:30 innoabis-71 nomad[3322]: 2019-04-24T15:08:30.049+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 15:08:34 innoabis-71 nomad[3322]: 2019-04-24T15:08:34.196+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:35 innoabis-71 nomad[3322]: 2019-04-24T15:08:35.416+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:36 innoabis-71 nomad[3322]: 2019-04-24T15:08:36.395+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:08:37 innoabis-71 nomad[3322]: 2019-04-24T15:08:37.682+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:17:51 innoabis-71 nomad[3322]: 2019-04-24T15:17:51.591+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:17:59 innoabis-71 nomad[3322]: 2019-04-24T15:17:59.924+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:18:07 innoabis-71 nomad[3322]: 2019-04-24T15:18:07.556+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/e0d67b3c-1aa5-967f-bc24-d42bf64744a5?follow=true&offset=50000&origin=end&task=WebAFIS-Delayed-Job&type=stdout error="failed to list entries: open
Apr 24 15:18:07 innoabis-71 nomad[3322]: 2019-04-24T15:18:07.562+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/e0d67b3c-1aa5-967f-bc24-d42bf64744a5?follow=true&offset=50000&origin=end&task=WebAFIS-Delayed-Job&type=stderr error="failed to list entries: open
Apr 24 15:18:08 innoabis-71 nomad[3322]: 2019-04-24T15:18:08.397+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/e0d67b3c-1aa5-967f-bc24-d42bf64744a5?follow=true&offset=50000&origin=end&task=WebAFIS-Delayed-Job&type=stdout error="failed to list entries: open
Apr 24 15:18:14 innoabis-71 nomad[3322]: 2019-04-24T15:18:14.746+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/e0d67b3c-1aa5-967f-bc24-d42bf64744a5?follow=true&offset=50000&origin=end&task=WebAFIS-Hitlist-Worker&type=stdout error="failed to list entries: op
Apr 24 15:18:19 innoabis-71 nomad[3322]: 2019-04-24T15:18:19.721+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:18:21 innoabis-71 nomad[3322]: 2019-04-24T15:18:21.340+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.494+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/16f7e5e1-5f
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.494+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Webserver path=/var/lib/nomad/alloc/16f7e5e1-5f3c-086e-906c-54
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.744+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.757+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/16f7
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.757+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/16f7
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.813+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/16f7e5
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.813+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/16f7e5
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.833+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/16f7e5
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.833+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/16f7e5
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.886+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/16f7e5e1-
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.886+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/16f7e5e1-
Apr 24 15:18:23 innoabis-71 nomad[3322]: 2019-04-24T15:18:23.984+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_websocket_serve
Apr 24 15:18:24 innoabis-71 nomad[3322]: 2019-04-24T15:18:24.172+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_worker_mq.sh --
Apr 24 15:18:24 innoabis-71 nomad[3322]: 2019-04-24T15:18:24.244+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webafis/hitlist_worker && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 24 15:18:24 innoabis-71 nomad[3322]: 2019-04-24T15:18:24.290+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_delayed_job.sh]
Apr 24 15:18:34 innoabis-71 nomad[3322]: 2019-04-24T15:18:34.653+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:18:35 innoabis-71 nomad[3322]: 2019-04-24T15:18:35.371+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:19:29 innoabis-71 nomad[3322]: 2019-04-24T15:19:29.540+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:19:32 innoabis-71 nomad[3322]: 2019-04-24T15:19:32.266+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:19:39 innoabis-71 nomad[3322]: 2019-04-24T15:19:39.489+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:20:19 innoabis-71 nomad[3322]: 2019-04-24T15:20:19.246+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:20:24 innoabis-71 nomad[3322]: 2019-04-24T15:20:24.689+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:21:23 innoabis-71 nomad[3322]: 2019-04-24T15:21:23.629+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:21:24 innoabis-71 nomad[3322]: 2019-04-24T15:21:24.011+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 15:21:54 innoabis-71 nomad[3322]: 2019-04-24T15:21:54.899+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/ffd88753-3f61-3fad-4652-c6347a4678e7?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /v
Apr 24 15:21:57 innoabis-71 nomad[3322]: 2019-04-24T15:21:57.328+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/ffd88753-3f61-3fad-4652-c6347a4678e7?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /v
Apr 24 15:21:57 innoabis-71 nomad[3322]: 2019-04-24T15:21:57.963+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/ffd88753-3f61-3fad-4652-c6347a4678e7?follow=true&offset=50000&origin=end&task=Message-Processor&type=stderr error="failed to list entries: open /v
Apr 24 15:22:04 innoabis-71 nomad[3322]: 2019-04-24T15:22:04.858+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/ffd88753-3f61-3fad-4652-c6347a4678e7?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /v
Apr 24 15:22:09 innoabis-71 nomad[3322]: 2019-04-24T15:22:09.196+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/ffd88753-3f61-3fad-4652-c6347a4678e7?follow=true&offset=50000&origin=end&task=Message-Processor&type=stderr error="failed to list entries: open /v
Apr 24 15:22:59 innoabis-71 nomad[3322]: 2019-04-24T15:22:59.212+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 15:23:23 innoabis-71 nomad[3322]: 2019-04-24T15:23:23.973+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/bd88dbd7-a073-a078-2677-48f5fc0ac575?follow=true&offset=50000&origin=end&task=INode&type=stdout error="failed to list entries: open /var/lib/innov
Apr 24 15:23:26 innoabis-71 nomad[3322]: 2019-04-24T15:23:26.482+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/bd88dbd7-a073-a078-2677-48f5fc0ac575?follow=true&offset=50000&origin=end&task=INode&type=stderr error="failed to list entries: open /var/lib/innov
Apr 24 15:23:27 innoabis-71 nomad[3322]: 2019-04-24T15:23:27.226+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/bd88dbd7-a073-a078-2677-48f5fc0ac575?follow=true&offset=50000&origin=end&task=INode&type=stdout error="failed to list entries: open /var/lib/innov
Apr 24 15:23:28 innoabis-71 nomad[3322]: 2019-04-24T15:23:28.300+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/bd88dbd7-a073-a078-2677-48f5fc0ac575?follow=true&offset=50000&origin=end&task=INode&type=stderr error="failed to list entries: open /var/lib/innov
Apr 24 15:23:29 innoabis-71 nomad[3322]: 2019-04-24T15:23:29.113+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/bd88dbd7-a073-a078-2677-48f5fc0ac575?follow=true&offset=50000&origin=end&task=INode&type=stdout error="failed to list entries: open /var/lib/innov
Apr 24 15:23:31 innoabis-71 nomad[3322]: 2019-04-24T15:23:31.847+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/bd88dbd7-a073-a078-2677-48f5fc0ac575?follow=true&offset=50000&origin=end&task=INode&type=stdout error="failed to list entries: open /var/lib/innov
Apr 24 15:23:51 innoabis-71 nomad[3322]: 2019-04-24T15:23:51.213+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 15:23:53 innoabis-71 nomad[3322]: 2019-04-24T15:23:53.321+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 15:24:34 innoabis-71 nomad[3322]: 2019-04-24T15:24:34.102+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stderr error="failed to list entries: open /var/li
Apr 24 15:24:34 innoabis-71 nomad[3322]: 2019-04-24T15:24:34.672+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 15:24:50 innoabis-71 nomad[3322]: 2019-04-24T15:24:50.173+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/337e2138-5084-635a-2752-d119c7640b2a?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 15:24:51 innoabis-71 nomad[3322]: 2019-04-24T15:24:51.000+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/337e2138-5084-635a-2752-d119c7640b2a?follow=true&offset=50000&origin=end&task=Fast-Service&type=stderr error="failed to list entries: open /var/li
Apr 24 15:25:31 innoabis-71 nomad[3322]: 2019-04-24T15:25:31.450+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/67b0a2cd-2072-9a0c-8068-f3661226b680?follow=true&offset=50000&origin=end&task=Fast-Service&type=stdout error="failed to list entries: open /var/li
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.169+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=337e2138-5084-635a-2752-d119c7640b2a reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=996081a7-5642-6ad8-2e73-057b3e737403 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=052e2a63-d1dd-293d-9265-ccf70af45498 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f70f1336-e523-95a1-88ae-d1b2f2e09734 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=a19db4cc-e6ec-5cdf-a2c6-912d524b253d reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f64e09aa-7425-70b4-e770-c158c03285fd reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=ffd88753-3f61-3fad-4652-c6347a4678e7 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=762568ec-0ca1-dd25-0f5b-84b404aa672b reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=0839351f-0605-1086-db26-a65779e5b111 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2748dd53-e986-dbd1-0e20-ed21c847c418 reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=866c09df-221c-0f94-db69-58e8e4b535ee reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.171+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=58b7ebbb-9761-7a2c-d434-f163a7dbedde reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=64978202-6ca0-df9b-7af8-4f3ff010b47d reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=41aa95e6-5f8d-b69b-9576-bcc1752e0a3a reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.170+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=03cf15be-fbd0-72dc-0a41-688ed3a5c96f reason="forced collection"
Apr 24 16:01:50 innoabis-71 nomad[3322]: 2019-04-24T16:01:50.171+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 reason="forced collection"
Apr 24 16:04:11 innoabis-71 nomad[3322]: 2019-04-24T16:04:11.517+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/6a285306-3876-e99
Apr 24 16:04:11 innoabis-71 nomad[3322]: 2019-04-24T16:04:11.517+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/6a285306-3876-e99
Apr 24 16:04:11 innoabis-71 nomad[3322]: 2019-04-24T16:04:11.523+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 16:04:13 innoabis-71 nomad[3322]: 2019-04-24T16:04:13.605+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=6a285306-3876-e991-e5d3-688669eed226 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 16:04:13 innoabis-71 nomad[3322]: 2019-04-24T16:04:13.606+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher reason="Restart within policy" delay=17.773865295s
Apr 24 16:04:31 innoabis-71 nomad[3322]: 2019-04-24T16:04:31.389+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/6a285306-3876-e99
Apr 24 16:04:31 innoabis-71 nomad[3322]: 2019-04-24T16:04:31.389+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/6a285306-3876-e99
Apr 24 16:04:31 innoabis-71 nomad[3322]: 2019-04-24T16:04:31.390+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 16:04:33 innoabis-71 nomad[3322]: 2019-04-24T16:04:33.477+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=6a285306-3876-e991-e5d3-688669eed226 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 16:04:33 innoabis-71 nomad[3322]: 2019-04-24T16:04:33.478+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=6a285306-3876-e991-e5d3-688669eed226 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 16:04:33 innoabis-71 nomad[3322]: 2019-04-24T16:04:33.478+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher reason="Restart within policy" delay=18.6047031s
Apr 24 16:04:52 innoabis-71 nomad[3322]: 2019-04-24T16:04:52.089+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher path=/var/lib/nomad/alloc/6a285306-3876-e991-e5d3-688669ee
Apr 24 16:04:52 innoabis-71 nomad[3322]: 2019-04-24T16:04:52.090+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/6a285306-3876-e99
Apr 24 16:04:52 innoabis-71 nomad[3322]: 2019-04-24T16:04:52.091+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 16:04:54 innoabis-71 nomad[3322]: 2019-04-24T16:04:54.161+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=6a285306-3876-e991-e5d3-688669eed226 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 16:04:54 innoabis-71 nomad[3322]: 2019-04-24T16:04:54.162+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=6a285306-3876-e991-e5d3-688669eed226 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 16:04:54 innoabis-71 nomad[3322]: 2019-04-24T16:04:54.168+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=6a285306-3876-e991-e5d3-688669eed226
Apr 24 16:21:50 innoabis-71 nomad[3322]: 2019-04-24T16:21:50.144+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=b50e56fa-91e1-8e3c-754d-49802c364a93 reason="forced collection"
Apr 24 17:04:54 innoabis-71 nomad[3322]: 2019-04-24T17:04:54.241+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/4e11ab8a-4424-338
Apr 24 17:04:54 innoabis-71 nomad[3322]: 2019-04-24T17:04:54.245+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/4e11ab8a-4424-338
Apr 24 17:04:54 innoabis-71 nomad[3322]: 2019-04-24T17:04:54.252+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 17:04:56 innoabis-71 nomad[3322]: 2019-04-24T17:04:56.344+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=4e11ab8a-4424-3383-b175-935081371347 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 17:04:56 innoabis-71 nomad[3322]: 2019-04-24T17:04:56.346+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher reason="Restart within policy" delay=15.390416903s
Apr 24 17:05:11 innoabis-71 nomad[3322]: 2019-04-24T17:05:11.740+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/4e11ab8a-4424-338
Apr 24 17:05:11 innoabis-71 nomad[3322]: 2019-04-24T17:05:11.740+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/4e11ab8a-4424-338
Apr 24 17:05:11 innoabis-71 nomad[3322]: 2019-04-24T17:05:11.741+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 17:05:13 innoabis-71 nomad[3322]: 2019-04-24T17:05:13.811+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=4e11ab8a-4424-3383-b175-935081371347 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 17:05:13 innoabis-71 nomad[3322]: 2019-04-24T17:05:13.812+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=4e11ab8a-4424-3383-b175-935081371347 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 17:05:13 innoabis-71 nomad[3322]: 2019-04-24T17:05:13.812+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher reason="Restart within policy" delay=17.706364839s
Apr 24 17:05:31 innoabis-71 nomad[3322]: 2019-04-24T17:05:31.522+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher path=/var/lib/nomad/alloc/4e11ab8a-4424-3383-b175-93508137
Apr 24 17:05:31 innoabis-71 nomad[3322]: 2019-04-24T17:05:31.523+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/4e11ab8a-4424-338
Apr 24 17:05:31 innoabis-71 nomad[3322]: 2019-04-24T17:05:31.523+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 17:05:32 innoabis-71 nomad[3322]: 2019-04-24T17:05:32.617+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=4e11ab8a-4424-3383-b175-935081371347 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 17:05:32 innoabis-71 nomad[3322]: 2019-04-24T17:05:32.617+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=4e11ab8a-4424-3383-b175-935081371347 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 17:05:32 innoabis-71 nomad[3322]: 2019-04-24T17:05:32.624+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=4e11ab8a-4424-3383-b175-935081371347
Apr 24 17:26:50 innoabis-71 nomad[3322]: 2019-04-24T17:26:50.144+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=75e56125-41cf-284d-a99e-3e99342deeae reason="forced collection"
Apr 24 18:05:32 innoabis-71 nomad[3322]: 2019-04-24T18:05:32.670+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/47dfa489-2674-593
Apr 24 18:05:32 innoabis-71 nomad[3322]: 2019-04-24T18:05:32.670+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher path=/var/lib/nomad/alloc/47dfa489-2674-5932-822a-bb249046
Apr 24 18:05:32 innoabis-71 nomad[3322]: 2019-04-24T18:05:32.691+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 18:05:34 innoabis-71 nomad[3322]: 2019-04-24T18:05:34.757+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=47dfa489-2674-5932-822a-bb249046222e driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 18:05:34 innoabis-71 nomad[3322]: 2019-04-24T18:05:34.758+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=47dfa489-2674-5932-822a-bb249046222e driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 18:05:34 innoabis-71 nomad[3322]: 2019-04-24T18:05:34.758+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher reason="Restart within policy" delay=17.210700715s
Apr 24 18:05:51 innoabis-71 nomad[3322]: 2019-04-24T18:05:51.976+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/47dfa489-2674-593
Apr 24 18:05:51 innoabis-71 nomad[3322]: 2019-04-24T18:05:51.977+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/47dfa489-2674-593
Apr 24 18:05:51 innoabis-71 nomad[3322]: 2019-04-24T18:05:51.977+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 18:05:54 innoabis-71 nomad[3322]: 2019-04-24T18:05:54.062+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=47dfa489-2674-5932-822a-bb249046222e driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 18:05:54 innoabis-71 nomad[3322]: 2019-04-24T18:05:54.064+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher reason="Restart within policy" delay=18.193841963s
Apr 24 18:06:12 innoabis-71 nomad[3322]: 2019-04-24T18:06:12.263+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/47dfa489-2674-593
Apr 24 18:06:12 innoabis-71 nomad[3322]: 2019-04-24T18:06:12.263+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/47dfa489-2674-593
Apr 24 18:06:12 innoabis-71 nomad[3322]: 2019-04-24T18:06:12.264+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 18:06:14 innoabis-71 nomad[3322]: 2019-04-24T18:06:14.334+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=47dfa489-2674-5932-822a-bb249046222e driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 18:06:14 innoabis-71 nomad[3322]: 2019-04-24T18:06:14.335+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 18:06:14 innoabis-71 nomad[3322]: 2019-04-24T18:06:14.341+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=47dfa489-2674-5932-822a-bb249046222e
Apr 24 18:06:14 innoabis-71 nomad[3322]: 2019-04-24T18:06:14.344+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=47dfa489-2674-5932-822a-bb249046222e task=IDispatcher error="read |0: file already closed"
Apr 24 18:31:50 innoabis-71 nomad[3322]: 2019-04-24T18:31:50.147+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=6a285306-3876-e991-e5d3-688669eed226 reason="forced collection"
Apr 24 18:34:17 innoabis-71 nomad[3322]: 2019-04-24T18:34:17.509+0200 [WARN ] nomad.heartbeat: node TTL expired: node_id=d2ff5fcf-7040-71cb-89d5-25c8a1ba4f35
Apr 24 18:34:18 innoabis-71 nomad[3322]: 2019-04-24T18:34:18.756+0200 [WARN ] client: missed heartbeat: req_latency=132.861664ms heartbeat_ttl=13.288839589s since_last_heartbeat=1.205265017s
Apr 24 18:34:18 innoabis-71 nomad[3322]: 2019-04-24T18:34:18.855+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-MQ-AFIS-Worker error="rpc error
Apr 24 18:34:18 innoabis-71 nomad[3322]: 2019-04-24T18:34:18.985+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Webserver error="rpc error: cod
Apr 24 18:34:18 innoabis-71 nomad[3322]: 2019-04-24T18:34:18.986+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Webserver error="read |0: file already closed"
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.000+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Websocket-Server error="rpc err
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.112+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Webserver error="read |0: file already closed"
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.167+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/8150fc43-79
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.167+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/8150fc43-79
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.175+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Websocket-Server path=/var/lib/nomad/alloc/8150fc43-79af-034b-
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.175+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/8150
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.201+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/8150fc43-
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.202+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Delayed-Job path=/var/lib/nomad/alloc/8150fc43-79af-034b-cf0c-
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.205+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.230+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_websocket_serve
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.234+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_delayed_job.sh]
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.239+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/8150fc
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.239+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-Hitlist-Worker path=/var/lib/nomad/alloc/8150fc43-79af-034b-cf
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.241+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/8150fc
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.241+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc task=WebAFIS-MQ-AFIS-Worker path=/var/lib/nomad/alloc/8150fc43-79af-034b-cf
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.256+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webafis/hitlist_worker && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 24 18:34:19 innoabis-71 nomad[3322]: 2019-04-24T18:34:19.310+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_worker_mq.sh --
Apr 24 18:34:22 innoabis-71 nomad[3322]: 2019-04-24T18:34:22.616+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Delayed-Job error="rpc error: c
Apr 24 18:34:22 innoabis-71 nomad[3322]: 2019-04-24T18:34:22.647+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Delayed-Job error="read |0: file already closed"
Apr 24 18:34:22 innoabis-71 nomad[3322]: 2019-04-24T18:34:22.699+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb task=WebAFIS-Delayed-Job error="read |0: file already closed"
Apr 24 18:34:23 innoabis-71 nomad[3322]: 2019-04-24T18:34:23.780+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Hitlist-Worker error="rpc error
Apr 24 18:34:23 innoabis-71 nomad[3322]: 2019-04-24T18:34:23.781+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb driver=raw_exec task_name=WebAFIS-Hitlist-Worker error="read |0: file already closed"
Apr 24 18:34:23 innoabis-71 nomad[3322]: 2019-04-24T18:34:23.826+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb
Apr 24 19:06:14 innoabis-71 nomad[3322]: 2019-04-24T19:06:14.398+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c2824d53-2296-7d6
Apr 24 19:06:14 innoabis-71 nomad[3322]: 2019-04-24T19:06:14.398+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c2824d53-2296-7d6
Apr 24 19:06:14 innoabis-71 nomad[3322]: 2019-04-24T19:06:14.407+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 19:06:16 innoabis-71 nomad[3322]: 2019-04-24T19:06:16.519+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 19:06:16 innoabis-71 nomad[3322]: 2019-04-24T19:06:16.520+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 19:06:16 innoabis-71 nomad[3322]: 2019-04-24T19:06:16.520+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher reason="Restart within policy" delay=16.459641997s
Apr 24 19:06:32 innoabis-71 nomad[3322]: 2019-04-24T19:06:32.983+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c2824d53-2296-7d6
Apr 24 19:06:32 innoabis-71 nomad[3322]: 2019-04-24T19:06:32.983+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher path=/var/lib/nomad/alloc/c2824d53-2296-7d6c-d6ad-4dd49501
Apr 24 19:06:32 innoabis-71 nomad[3322]: 2019-04-24T19:06:32.984+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 19:06:34 innoabis-71 nomad[3322]: 2019-04-24T19:06:34.059+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 19:06:34 innoabis-71 nomad[3322]: 2019-04-24T19:06:34.060+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 19:06:34 innoabis-71 nomad[3322]: 2019-04-24T19:06:34.060+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher reason="Restart within policy" delay=18.677225104s
Apr 24 19:06:52 innoabis-71 nomad[3322]: 2019-04-24T19:06:52.741+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c2824d53-2296-7d6
Apr 24 19:06:52 innoabis-71 nomad[3322]: 2019-04-24T19:06:52.741+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c2824d53-2296-7d6
Apr 24 19:06:52 innoabis-71 nomad[3322]: 2019-04-24T19:06:52.742+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 19:06:54 innoabis-71 nomad[3322]: 2019-04-24T19:06:54.806+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 19:06:54 innoabis-71 nomad[3322]: 2019-04-24T19:06:54.807+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 19:06:54 innoabis-71 nomad[3322]: 2019-04-24T19:06:54.811+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709
Apr 24 19:36:50 innoabis-71 nomad[3322]: 2019-04-24T19:36:50.152+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=4e11ab8a-4424-3383-b175-935081371347 reason="forced collection"
Apr 24 19:37:06 innoabis-71 nomad[3322]: 2019-04-24T19:37:06.747+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:37:06 innoabis-71 nomad[3322]: 2019-04-24T19:37:06.959+0200 [ERROR] http: request failed: method=GET path=/v1/job/matcher error="job not found" code=404
Apr 24 19:37:09 innoabis-71 nomad[3322]: 2019-04-24T19:37:09.740+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:37:16 innoabis-71 nomad[3322]: 2019-04-24T19:37:16.067+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 19:37:22 innoabis-71 nomad[3322]: 2019-04-24T19:37:22.534+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/4e11ab8a-4424-3383-b175-935081371347 error="alloc not found" code=404
Apr 24 19:37:22 innoabis-71 nomad[3322]: 2019-04-24T19:37:22.537+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/6a285306-3876-e991-e5d3-688669eed226 error="alloc not found" code=404
Apr 24 19:37:22 innoabis-71 nomad[3322]: 2019-04-24T19:37:22.539+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/75e56125-41cf-284d-a99e-3e99342deeae error="alloc not found" code=404
Apr 24 19:37:22 innoabis-71 nomad[3322]: 2019-04-24T19:37:22.541+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/b50e56fa-91e1-8e3c-754d-49802c364a93 error="alloc not found" code=404
Apr 24 19:37:22 innoabis-71 nomad[3322]: 2019-04-24T19:37:22.544+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/866c09df-221c-0f94-db69-58e8e4b535ee error="alloc not found" code=404
Apr 24 19:38:33 innoabis-71 nomad[3322]: 2019-04-24T19:38:33.435+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:38:34 innoabis-71 nomad[3322]: 2019-04-24T19:38:34.442+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:38:40 innoabis-71 nomad[3322]: 2019-04-24T19:38:40.961+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/996081a7-5642-6ad8-2e73-057b3e737403 error="alloc not found" code=404
Apr 24 19:38:41 innoabis-71 nomad[3322]: 2019-04-24T19:38:41.347+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 19:38:41 innoabis-71 nomad[3322]: 2019-04-24T19:38:41.350+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/996081a7-5642-6ad8-2e73-057b3e737403 error="alloc not found" code=404
Apr 24 19:43:29 innoabis-71 nomad[3322]: 2019-04-24T19:43:29.427+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:43:29 innoabis-71 nomad[3322]: 2019-04-24T19:43:29.646+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 19:43:31 innoabis-71 nomad[3322]: 2019-04-24T19:43:31.425+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:43:31 innoabis-71 nomad[3322]: 2019-04-24T19:43:31.786+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 19:47:11 innoabis-71 nomad[3322]: 2019-04-24T19:47:11.499+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 24 19:47:11 innoabis-71 nomad[3322]: 2019-04-24T19:47:11.803+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 24 20:06:50 innoabis-71 nomad[3322]: 2019-04-24T20:06:50.141+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=16f7e5e1-5f3c-086e-906c-54c994707dfb reason="forced collection"
Apr 24 20:06:54 innoabis-71 nomad[3322]: 2019-04-24T20:06:54.871+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/181ecf81-8947-406
Apr 24 20:06:54 innoabis-71 nomad[3322]: 2019-04-24T20:06:54.871+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/181ecf81-8947-406
Apr 24 20:06:54 innoabis-71 nomad[3322]: 2019-04-24T20:06:54.879+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 20:06:56 innoabis-71 nomad[3322]: 2019-04-24T20:06:56.959+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 20:06:56 innoabis-71 nomad[3322]: 2019-04-24T20:06:56.960+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 20:06:56 innoabis-71 nomad[3322]: 2019-04-24T20:06:56.960+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher reason="Restart within policy" delay=17.938138462s
Apr 24 20:07:14 innoabis-71 nomad[3322]: 2019-04-24T20:07:14.905+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/181ecf81-8947-406
Apr 24 20:07:14 innoabis-71 nomad[3322]: 2019-04-24T20:07:14.905+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/181ecf81-8947-406
Apr 24 20:07:14 innoabis-71 nomad[3322]: 2019-04-24T20:07:14.905+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 20:07:16 innoabis-71 nomad[3322]: 2019-04-24T20:07:16.987+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 20:07:16 innoabis-71 nomad[3322]: 2019-04-24T20:07:16.990+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher reason="Restart within policy" delay=17.843917564s
Apr 24 20:07:34 innoabis-71 nomad[3322]: 2019-04-24T20:07:34.843+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/181ecf81-8947-406
Apr 24 20:07:34 innoabis-71 nomad[3322]: 2019-04-24T20:07:34.843+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/181ecf81-8947-406
Apr 24 20:07:34 innoabis-71 nomad[3322]: 2019-04-24T20:07:34.845+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 20:07:36 innoabis-71 nomad[3322]: 2019-04-24T20:07:36.937+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 20:07:36 innoabis-71 nomad[3322]: 2019-04-24T20:07:36.940+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 20:07:36 innoabis-71 nomad[3322]: 2019-04-24T20:07:36.940+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 20:07:36 innoabis-71 nomad[3322]: 2019-04-24T20:07:36.949+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a
Apr 24 20:07:36 innoabis-71 nomad[3322]: 2019-04-24T20:07:36.958+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=181ecf81-8947-4068-6820-f42f12f7b65a task=IDispatcher error="read |0: file already closed"
Apr 24 20:41:50 innoabis-71 nomad[3322]: 2019-04-24T20:41:50.157+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=47dfa489-2674-5932-822a-bb249046222e reason="forced collection"
Apr 24 21:07:37 innoabis-71 nomad[3322]: 2019-04-24T21:07:37.028+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-356
Apr 24 21:07:37 innoabis-71 nomad[3322]: 2019-04-24T21:07:37.028+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-356
Apr 24 21:07:37 innoabis-71 nomad[3322]: 2019-04-24T21:07:37.038+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 21:07:39 innoabis-71 nomad[3322]: 2019-04-24T21:07:39.129+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 21:07:39 innoabis-71 nomad[3322]: 2019-04-24T21:07:39.131+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher reason="Restart within policy" delay=17.20800864s
Apr 24 21:07:56 innoabis-71 nomad[3322]: 2019-04-24T21:07:56.348+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-356
Apr 24 21:07:56 innoabis-71 nomad[3322]: 2019-04-24T21:07:56.348+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-356
Apr 24 21:07:56 innoabis-71 nomad[3322]: 2019-04-24T21:07:56.349+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 21:07:58 innoabis-71 nomad[3322]: 2019-04-24T21:07:58.419+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 21:07:58 innoabis-71 nomad[3322]: 2019-04-24T21:07:58.420+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 21:07:58 innoabis-71 nomad[3322]: 2019-04-24T21:07:58.420+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher reason="Restart within policy" delay=16.976791174s
Apr 24 21:08:15 innoabis-71 nomad[3322]: 2019-04-24T21:08:15.404+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-356
Apr 24 21:08:15 innoabis-71 nomad[3322]: 2019-04-24T21:08:15.407+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-356
Apr 24 21:08:15 innoabis-71 nomad[3322]: 2019-04-24T21:08:15.409+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.513+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.513+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.519+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.526+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher error="read |0: file already closed"
Apr 24 21:46:50 innoabis-71 nomad[3322]: 2019-04-24T21:46:50.156+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 reason="forced collection"
Apr 24 22:08:17 innoabis-71 nomad[3322]: 2019-04-24T22:08:17.604+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d
Apr 24 22:08:17 innoabis-71 nomad[3322]: 2019-04-24T22:08:17.605+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae7
Apr 24 22:08:17 innoabis-71 nomad[3322]: 2019-04-24T22:08:17.614+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 22:08:19 innoabis-71 nomad[3322]: 2019-04-24T22:08:19.755+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 22:08:19 innoabis-71 nomad[3322]: 2019-04-24T22:08:19.756+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher reason="Restart within policy" delay=16.873270727s
Apr 24 22:08:36 innoabis-71 nomad[3322]: 2019-04-24T22:08:36.634+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae7
Apr 24 22:08:36 innoabis-71 nomad[3322]: 2019-04-24T22:08:36.634+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae7
Apr 24 22:08:36 innoabis-71 nomad[3322]: 2019-04-24T22:08:36.636+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 22:08:37 innoabis-71 nomad[3322]: 2019-04-24T22:08:37.750+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 22:08:37 innoabis-71 nomad[3322]: 2019-04-24T22:08:37.751+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 22:08:37 innoabis-71 nomad[3322]: 2019-04-24T22:08:37.751+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher reason="Restart within policy" delay=18.711939841s
Apr 24 22:08:56 innoabis-71 nomad[3322]: 2019-04-24T22:08:56.471+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae7
Apr 24 22:08:56 innoabis-71 nomad[3322]: 2019-04-24T22:08:56.471+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae7
Apr 24 22:08:56 innoabis-71 nomad[3322]: 2019-04-24T22:08:56.473+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 22:08:58 innoabis-71 nomad[3322]: 2019-04-24T22:08:58.581+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 24 22:08:58 innoabis-71 nomad[3322]: 2019-04-24T22:08:58.584+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 22:08:58 innoabis-71 nomad[3322]: 2019-04-24T22:08:58.591+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8
Apr 24 22:11:28 innoabis-71 nomad[3322]: 2019-04-24T22:11:28.600+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1194 error="alloc not found" code=404
Apr 24 22:16:56 innoabis-71 nomad[3322]: 2019-04-24T22:16:56.349+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1461 error="alloc not found" code=404
Apr 24 22:23:35 innoabis-71 nomad[3322]: 2019-04-24T22:23:35.798+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /v
Apr 24 22:23:43 innoabis-71 nomad[3322]: 2019-04-24T22:23:43.565+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stderr error="failed to list entries: open /v
Apr 24 22:23:54 innoabis-71 nomad[3322]: 2019-04-24T22:23:54.517+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /v
Apr 24 22:23:57 innoabis-71 nomad[3322]: 2019-04-24T22:23:57.827+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /v
Apr 24 22:23:58 innoabis-71 nomad[3322]: 2019-04-24T22:23:58.563+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?offset=0&origin=start&plain=true&task=Message-Processor&type=stdout error="failed to list entries: open /var/
Apr 24 22:29:24 innoabis-71 nomad[3322]: 2019-04-24T22:29:24.372+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1461 error="alloc not found" code=404
Apr 24 22:38:10 innoabis-71 nomad[3322]: 2019-04-24T22:38:10.177+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1461 error="alloc not found" code=404
lines 537-578/578 (END)
Apr 24 21:07:39 innoabis-71 nomad[3322]: 2019-04-24T21:07:39.129+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 24 21:07:39 innoabis-71 nomad[3322]: 2019-04-24T21:07:39.131+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher reason="Restart within policy" delay=17.20800864s
Apr 24 21:07:56 innoabis-71 nomad[3322]: 2019-04-24T21:07:56.348+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-3568-0f82-49a426bc98ec/alloc/logs/.IDis
Apr 24 21:07:56 innoabis-71 nomad[3322]: 2019-04-24T21:07:56.348+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-3568-0f82-49a426bc98ec/alloc/logs/.IDis
Apr 24 21:07:56 innoabis-71 nomad[3322]: 2019-04-24T21:07:56.349+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 21:07:58 innoabis-71 nomad[3322]: 2019-04-24T21:07:58.419+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 24 21:07:58 innoabis-71 nomad[3322]: 2019-04-24T21:07:58.420+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 21:07:58 innoabis-71 nomad[3322]: 2019-04-24T21:07:58.420+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher reason="Restart within policy" delay=16.976791174s
Apr 24 21:08:15 innoabis-71 nomad[3322]: 2019-04-24T21:08:15.404+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-3568-0f82-49a426bc98ec/alloc/logs/.IDis
Apr 24 21:08:15 innoabis-71 nomad[3322]: 2019-04-24T21:08:15.407+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/7797f96b-4ebc-3568-0f82-49a426bc98ec/alloc/logs/.IDis
Apr 24 21:08:15 innoabis-71 nomad[3322]: 2019-04-24T21:08:15.409+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.513+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.513+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.519+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec
Apr 24 21:08:17 innoabis-71 nomad[3322]: 2019-04-24T21:08:17.526+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec task=IDispatcher error="read |0: file already closed"
Apr 24 21:46:50 innoabis-71 nomad[3322]: 2019-04-24T21:46:50.156+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=c2824d53-2296-7d6c-d6ad-4dd495016709 reason="forced collection"
Apr 24 22:08:17 innoabis-71 nomad[3322]: 2019-04-24T22:08:17.604+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d90b8/alloc/logs/.IDispatcher.stdout.
Apr 24 22:08:17 innoabis-71 nomad[3322]: 2019-04-24T22:08:17.605+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d90b8/alloc/logs/.IDis
Apr 24 22:08:17 innoabis-71 nomad[3322]: 2019-04-24T22:08:17.614+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 22:08:19 innoabis-71 nomad[3322]: 2019-04-24T22:08:19.755+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 24 22:08:19 innoabis-71 nomad[3322]: 2019-04-24T22:08:19.756+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher reason="Restart within policy" delay=16.873270727s
Apr 24 22:08:36 innoabis-71 nomad[3322]: 2019-04-24T22:08:36.634+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d90b8/alloc/logs/.IDis
Apr 24 22:08:36 innoabis-71 nomad[3322]: 2019-04-24T22:08:36.634+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d90b8/alloc/logs/.IDis
Apr 24 22:08:36 innoabis-71 nomad[3322]: 2019-04-24T22:08:36.636+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 22:08:37 innoabis-71 nomad[3322]: 2019-04-24T22:08:37.750+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 24 22:08:37 innoabis-71 nomad[3322]: 2019-04-24T22:08:37.751+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 24 22:08:37 innoabis-71 nomad[3322]: 2019-04-24T22:08:37.751+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher reason="Restart within policy" delay=18.711939841s
Apr 24 22:08:56 innoabis-71 nomad[3322]: 2019-04-24T22:08:56.471+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d90b8/alloc/logs/.IDis
Apr 24 22:08:56 innoabis-71 nomad[3322]: 2019-04-24T22:08:56.471+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/c5d45bea-cc68-ae73-8319-b8960f2d90b8/alloc/logs/.IDis
Apr 24 22:08:56 innoabis-71 nomad[3322]: 2019-04-24T22:08:56.473+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 24 22:08:58 innoabis-71 nomad[3322]: 2019-04-24T22:08:58.581+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 24 22:08:58 innoabis-71 nomad[3322]: 2019-04-24T22:08:58.584+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 24 22:08:58 innoabis-71 nomad[3322]: 2019-04-24T22:08:58.591+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8
Apr 24 22:11:28 innoabis-71 nomad[3322]: 2019-04-24T22:11:28.600+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1194 error="alloc not found" code=404
Apr 24 22:16:56 innoabis-71 nomad[3322]: 2019-04-24T22:16:56.349+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1461 er.ror="alloc not found" code=404
Apr 24 22:23:35 innoabis-71 nomad[3322]: 2019-04-24T22:23:35.798+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /var/lib/nomad/alloc/f30a8
Apr 24 22:23:43 innoabis-71 nomad[3322]: 2019-04-24T22:23:43.565+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stderr error="failed to list entries: open /var/lib/nomad/alloc/f30a8
Apr 24 22:23:54 innoabis-71 nomad[3322]: 2019-04-24T22:23:54.517+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /var/lib/nomad/alloc/f30a8
Apr 24 22:23:57 innoabis-71 nomad[3322]: 2019-04-24T22:23:57.827+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?follow=true&offset=50000&origin=end&task=Message-Processor&type=stdout error="failed to list entries: open /var/lib/nomad/alloc/f30a8
Apr 24 22:23:58 innoabis-71 nomad[3322]: 2019-04-24T22:23:58.563+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/f30a8688-ef0d-6344-a8f2-a793911b14bb?offset=0&origin=start&plain=true&task=Message-Processor&type=stdout error="failed to list entries: open /var/lib/nomad/alloc/f30a8688
Apr 24 22:29:24 innoabis-71 nomad[3322]: 2019-04-24T22:29:24.372+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1461 error="alloc not found" code=404
Apr 24 22:38:10 innoabis-71 nomad[3322]: 2019-04-24T22:38:10.177+0200 [ERROR] http: request failed: method=GET path=/v1/allocation/16f7e5e1-5f3c-086e-906c-54c994707dfb?index=1461 error="alloc not found" code=404

v 0.9.1-rc1:

journalctl -u nomad
-- Logs begin at Thu 2019-04-25 14:52:37 CEST, end at Thu 2019-04-25 14:54:31 CEST. --
Apr 25 14:52:48 innoabis-71 systemd[1]: Started HashiCorp Nomad.
Apr 25 14:52:49 innoabis-71 nomad[3314]: WARNING: keyring exists but -encrypt given, using keyring
Apr 25 14:52:49 innoabis-71 nomad[3314]: ==> WARNING: Bootstrap mode enabled! Potentially unsafe operation.
Apr 25 14:52:49 innoabis-71 nomad[3314]: ==> Loaded configuration from /etc/nomad.d/nomad-config.hcl, /etc/nomad.d/nomad.hcl
Apr 25 14:52:49 innoabis-71 nomad[3314]: ==> Starting Nomad agent...
Apr 25 14:52:52 innoabis-71 nomad[3314]: ==> Nomad agent configuration:
Apr 25 14:52:52 innoabis-71 nomad[3314]: Advertise Addrs: HTTP: 10.0.2.15:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648
Apr 25 14:52:52 innoabis-71 nomad[3314]: Bind Addrs: HTTP: 0.0.0.0:4646; RPC: 10.0.2.15:4647; Serf: 10.0.2.15:4648
Apr 25 14:52:52 innoabis-71 nomad[3314]: Client: true
Apr 25 14:52:52 innoabis-71 nomad[3314]: Log Level: INFO
Apr 25 14:52:52 innoabis-71 nomad[3314]: Region: global (DC: dc1)
Apr 25 14:52:52 innoabis-71 nomad[3314]: Server: true
Apr 25 14:52:52 innoabis-71 nomad[3314]: Version: 0.9.1-rc1
Apr 25 14:52:52 innoabis-71 nomad[3314]: ==> Nomad agent started! Log data will stream in below:
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.742+0200 [WARN ] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/var/lib/nomad/plugins
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=nvidia-gpu type=device plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=rkt type=driver plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:49.778+0200 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.169+0200 [INFO ] nomad: raft: Initial configuration (index=1): [{Suffrage:Voter ID:10.0.2.15:4647 Address:10.0.2.15:4647}]
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.170+0200 [INFO ] nomad: raft: Node at 10.0.2.15:4647 [Follower] entering Follower state (Leader: "")
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.179+0200 [INFO ] nomad: serf: EventMemberJoin: innoabis-71.global 10.0.2.15
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.179+0200 [INFO ] nomad: starting scheduling worker(s): num_workers=2 schedulers="[service batch system _core]"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.181+0200 [WARN ] nomad: serf: Failed to re-join any previously known node
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.181+0200 [INFO ] nomad: adding server: server="innoabis-71.global (Addr: 10.0.2.15:4647) (DC: dc1)"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.183+0200 [INFO ] client: using state directory: state_dir=/var/lib/nomad/client
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.184+0200 [ERROR] nomad: error looking up Nomad servers in Consul: error="server.nomad: unable to query Consul datacenters: Get http://127.0.0.1:8500/v1/catalog/datacenters: dial tcp 127.0.0.1:8500: connect: connection re
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.198+0200 [INFO ] client: using alloc directory: alloc_dir=/var/lib/nomad/alloc
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.220+0200 [INFO ] client.fingerprint_mgr.cgroup: cgroups are available
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.232+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=driver
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.232+0200 [INFO ] client.plugin: starting plugin manager: plugin-type=device
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.510+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=3abfcd2d-49d6-4989-48a3-1432e788f
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.519+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=b89eaeab-7250-040b-3881-970baacb6
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.519+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=b89eaeab-7250-040b-3881-970baacb685e task=activemq515 error="failed to reattach to executor: error creating rpc client for ex
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.523+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=db2a4831-9953-cfa6-3b9f-ac4a80470
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.523+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver error="failed to reattach to executor: error creating rpc client
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.526+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=db2a4831-9953-cfa6-3b9f-ac4a80470
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.526+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-MQ-AFIS-Worker error="failed to reattach to executor: error creating rpc cl
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.531+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=db2a4831-9953-cfa6-3b9f-ac4a80470
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.531+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job error="failed to reattach to executor: error creating rpc clien
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.539+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=db2a4831-9953-cfa6-3b9f-ac4a80470
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.539+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker error="failed to reattach to executor: error creating rpc cl
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.541+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=db2a4831-9953-cfa6-3b9f-ac4a80470
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.541+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Websocket-Server error="failed to reattach to executor: error creating rpc
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.544+0200 [ERROR] client.driver_mgr.raw_exec: failed to reattach to executor: driver=raw_exec error="error creating rpc client for executor plugin: Reattachment process not found" task_id=f897d1f4-c260-bdb9-b698-8070397ee
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.544+0200 [ERROR] client.alloc_runner.task_runner: error recovering task; cleaning up: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 task=INode error="failed to reattach to executor: error creating rpc client for executor
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.544+0200 [INFO ] client: started client: node_id=d2ff5fcf-7040-71cb-89d5-25c8a1ba4f35
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.558+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Websocket-Server error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.577+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=b89eaeab-7250-040b-3881-970baacb685e task=activemq515 error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.586+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.593+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.617+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 task=INode error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.619+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-MQ-AFIS-Worker error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.627+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.628+0200 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker error="Reattachment process not found"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=3abfcd2d-49d6-4989-48a3-1432e788f845
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=abc490f7-b0d5-f4f0-d673-d73489c029f9
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=bfad63e1-2a86-f24f-a1a6-b11533f7074f
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=6854d42b-a643-0df8-5875-41f8f60537f2
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=b89eaeab-7250-040b-3881-970baacb685e
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.636+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.637+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.637+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=3abfcd2d-49d6-4989-48a3-1432e788f845 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=abc490f7-b0d5-f4f0-d673-d73489c029f9 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=bfad63e1-2a86-f24f-a1a6-b11533f7074f reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=6854d42b-a643-0df8-5875-41f8f60537f2 reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=b89eaeab-7250-040b-3881-970baacb685e reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.662+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae reason="forced collection"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.691+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Websocket-Server path=/var/lib/nomad/alloc/db2a4831-9953-cfa6-
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.691+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/db2a
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.825+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.826+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.826+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=b89eaeab-7250-040b-3881-970baacb685e task=activemq515 @module=logmon path=/var/lib/nomad/alloc/b89eaeab-7250-040
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.826+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=b89eaeab-7250-040b-3881-970baacb685e task=activemq515 @module=logmon path=/var/lib/nomad/alloc/b89eaeab-7250-040
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.866+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver path=/var/lib/nomad/alloc/db2a4831-9953-cfa6-3b9f-ac
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.866+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/db2a4831-99
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.885+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/db2a4831-
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.885+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/db2a4831-
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.890+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_websocket_serve
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.902+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_worker_mq.sh --
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.908+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.937+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.997+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 task=INode path=/var/lib/nomad/alloc/f897d1f4-c260-bdb9-b698-8070397ee9b8/a
Apr 25 14:52:52 innoabis-71 nomad[3314]: 2019-04-25T14:52:52.997+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 task=INode path=/var/lib/nomad/alloc/f897d1f4-c260-bdb9-b698-8070397ee9b8/a
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.003+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.003+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker path=/var/lib/nomad/alloc/db2a4831-9953-cfa6-3b
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.013+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_delayed_job.sh]
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.049+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./INode -c inode-config.xml -d 1]}"
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.077+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webafis/hitlist_worker && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.344+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/d0a12c58-e706
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.344+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/d0a12c58-e706
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.364+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/ddc91527-a1
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.364+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/ddc91527-a1
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.602+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /srv/apps/eafis/eafis-cluster/eafis.sh]}"
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.737+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 driver=raw_exec task_name=INode error="rpc error: code = Unavaila
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.831+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8 task=INode error="read |0: file already closed"
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.993+0200 [WARN ] nomad: raft: Heartbeat timeout from "" reached, starting election
Apr 25 14:52:53 innoabis-71 nomad[3314]: 2019-04-25T14:52:53.993+0200 [INFO ] nomad: raft: Node at 10.0.2.15:4647 [Candidate] entering Candidate state in term 17
Apr 25 14:52:54 innoabis-71 nomad[3314]: 2019-04-25T14:52:54.055+0200 [INFO ] nomad: raft: Election won. Tally: 1
Apr 25 14:52:54 innoabis-71 nomad[3314]: 2019-04-25T14:52:54.055+0200 [INFO ] nomad: raft: Node at 10.0.2.15:4647 [Leader] entering Leader state
Apr 25 14:52:54 innoabis-71 nomad[3314]: 2019-04-25T14:52:54.057+0200 [INFO ] nomad: cluster leadership acquired
Apr 25 14:52:54 innoabis-71 nomad[3314]: 2019-04-25T14:52:54.378+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=b89eaeab-7250-040b-3881-970baacb685e driver=raw_exec task_name=activemq515 error="rpc error: code = Un
Apr 25 14:52:54 innoabis-71 nomad[3314]: 2019-04-25T14:52:54.544+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/5b8a48f3-b817-0b0
Apr 25 14:52:54 innoabis-71 nomad[3314]: 2019-04-25T14:52:54.544+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 task=IDispatcher path=/var/lib/nomad/alloc/5b8a48f3-b817-0b0e-9223-c5309643
Apr 25 14:52:55 innoabis-71 nomad[3314]: 2019-04-25T14:52:55.209+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 task=INode path=/var/lib/nomad/alloc/e07196e7-5dea-ca85-63a4-97dfb3f25554/a
Apr 25 14:52:55 innoabis-71 nomad[3314]: 2019-04-25T14:52:55.210+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 task=INode @module=logmon path=/var/lib/nomad/alloc/e07196e7-5dea-ca85-63a4
Apr 25 14:52:55 innoabis-71 nomad[3314]: 2019-04-25T14:52:55.211+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/d0a12c58-e706
Apr 25 14:52:55 innoabis-71 nomad[3314]: 2019-04-25T14:52:55.211+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/d0a12c58-e706
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.072+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.311+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 @module=logmon path=/var/lib/nomad/alloc/2d7547bd-e955-70b
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.311+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 task=activemq515 @module=logmon path=/var/lib/nomad/alloc/2d7547bd-e955-70b
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.387+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=64978202-6ca0-df9b-7af8-4f3ff010b47d
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.402+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=337e2138-5084-635a-2752-d119c7640b2a task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/337e2138-5084-63
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.402+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=337e2138-5084-635a-2752-d119c7640b2a task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/337e2138-5084-63
Apr 25 14:52:56 innoabis-71 nomad[3314]: 2019-04-25T14:52:56.745+0200 [INFO ] client: node registration complete
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.221+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./INode -c inode-config.xml -d 1]}"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.245+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[/opt/abis-monitoring/bin/abis-monitoring.sh]}"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.374+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=58b7ebbb-9761-7a2c-d434-f163a7dbedde
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.375+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=052e2a63-d1dd-293d-9265-ccf70af45498
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.402+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.662+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.663+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.673+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-Webserver error="rpc error: cod
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.686+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=866c09df-221c-0f94-db69-58e8e4b535ee
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.686+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.739+0200 [ERROR] client.alloc_runner.task_runner: running driver failed: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring error="failed to create executor: error creating rpc client for executor plugin:
Apr 25 14:52:57 innoabis-71 nomad[3314]: This usually means that the plugin is either invalid or simply
Apr 25 14:52:57 innoabis-71 nomad[3314]: needs to be recompiled to support the latest protocol."
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.739+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 driver=raw_exec task_name=ABIS-Monitoring error="read |0: file already closed"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.751+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.777+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.818+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-MQ-AFIS-Worker error="rpc error
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.823+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-MQ-AFIS-Worker error="read |0: file already closed"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.826+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 22277]}"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.875+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring reason="Error was unrecoverable"
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.898+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=41aa95e6-5f8d-b69b-9576-bcc1752e0a3a
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.937+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/e0d6
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.937+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/e0d6
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.939+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver path=/var/lib/nomad/alloc/e0d67b3c-1aa5-967f-bc24-d4
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.939+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver path=/var/lib/nomad/alloc/e0d67b3c-1aa5-967f-bc24-d4
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.975+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/e0d67b3c-
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.975+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/e0d67b3c-
Apr 25 14:52:57 innoabis-71 nomad[3314]: 2019-04-25T14:52:57.996+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=ffd88753-3f61-3fad-4652-c6347a4678e7
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.079+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.080+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker path=/var/lib/nomad/alloc/e0d67b3c-1aa5-967f-bc
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.115+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2748dd53-e986-dbd1-0e20-ed21c847c418
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.232+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=b50e56fa-91e1-8e3c-754d-49802c364a93
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.406+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.406+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/e0d67b
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.420+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.471+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_websocket_serve
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.532+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=0839351f-0605-1086-db26-a65779e5b111
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.538+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/1b9bfd2b-5e
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.538+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c task=Message-Processor path=/var/lib/nomad/alloc/1b9bfd2b-5ed1-cea5-23fc-05
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.605+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_delayed_job.sh]
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.678+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=a19db4cc-e6ec-5cdf-a2c6-912d524b253d
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.679+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=03cf15be-fbd0-72dc-0a41-688ed3a5c96f
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.718+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Webserver&type=stdout error="Unknown allocation "db2a4831-99
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.754+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f64e09aa-7425-70b4-e770-c158c03285fd
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.758+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_worker_mq.sh --
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.857+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f70f1336-e523-95a1-88ae-d1b2f2e09734
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.910+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 driver=raw_exec task_name=activemq515 error="rpc error: code = Un
Apr 25 14:52:58 innoabis-71 nomad[3314]: 2019-04-25T14:52:58.922+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 driver=raw_exec task_name=activemq515 error="read |0: file already closed"
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.013+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webafis/hitlist_worker && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.045+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /srv/apps/eafis/eafis-cluster/eafis.sh]}"
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.080+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-Websocket-Server error="rpc err
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.273+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.388+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.546+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=762568ec-0ca1-dd25-0f5b-84b404aa672b
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.699+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Websocket-Server&type=stdout error="Unknown allocation "db2a
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.977+0200 [WARN ] consul.sync: failed to update services in Consul: error="Put http://127.0.0.1:8500/v1/agent/service/deregister/_nomad-task-5m54llcv5qfchdgqyzor2ibkp6njp6hg: net/http: request canceled (Client.Timeout exc
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.118+0200 [WARN ] client.alloc_runner.runner_hook.alloc_health_watcher: error looking up Consul registrations for allocation: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae error="Get http://127.0.0.1:8500/v1/agent/service
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.153+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 @module=logmon path=/var/lib/nomad/alloc/2232a445-10cc-3ae
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.153+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 path=/var/lib/nomad/alloc/2232a445-10cc-3ae6-3056-92a9016f
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.191+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.306+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor path=/var/lib/nomad/alloc/f30a8688-ef0d-6344-a8f2-a7
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.306+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f30a8688-ef0d-6344-a8f2-a793911b14bb task=Message-Processor @module=logmon path=/var/lib/nomad/alloc/f30a8688-ef
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.617+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /srv/apps/eafis/eafis-cluster/eafis.sh]}"
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.802+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service path=/var/lib/nomad/alloc/67b0a2cd-2072-9a0c-8068-f366122
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.802+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=67b0a2cd-2072-9a0c-8068-f3661226b680 task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/67b0a2cd-2072-9a
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.823+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode @module=logmon path=/var/lib/nomad/alloc/bd88dbd7-a073-a078-2677
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.823+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=bd88dbd7-a073-a078-2677-48f5fc0ac575 task=INode @module=logmon path=/var/lib/nomad/alloc/bd88dbd7-a073-a078-2677
Apr 25 14:52:59 innoabis-71 nomad[3314]: 2019-04-25T14:52:59.897+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 31849]}"
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.086+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./INode -c inode-config.xml -d 1]}"
Apr 25 14:53:00 innoabis-71 nomad[3314]: 2019-04-25T14:53:00.831+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-MQ-AFIS-Worker&type=stdout error="Unknown allocation "db2a48
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.285+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.285+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: timed out waiting for read-side of process output pipe to close: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring @module=logmon
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.290+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=d0a12c58-e706-ad48-191f-6bbf40ecf631 task=ABIS-Monitoring error="read |0: file already closed"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.512+0200 [INFO ] client: node registration complete
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.569+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.569+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.569+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=337e2138-5084-635a-2752-d119c7640b2a
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.569+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.569+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.569+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=866c09df-221c-0f94-db69-58e8e4b535ee reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=ffd88753-3f61-3fad-4652-c6347a4678e7 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=a19db4cc-e6ec-5cdf-a2c6-912d524b253d reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=41aa95e6-5f8d-b69b-9576-bcc1752e0a3a reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f64e09aa-7425-70b4-e770-c158c03285fd reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=03cf15be-fbd0-72dc-0a41-688ed3a5c96f reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2d7547bd-e955-70bf-587e-fc5af971fd04 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=052e2a63-d1dd-293d-9265-ccf70af45498 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=762568ec-0ca1-dd25-0f5b-84b404aa672b reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=58b7ebbb-9761-7a2c-d434-f163a7dbedde reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=f70f1336-e523-95a1-88ae-d1b2f2e09734 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=337e2138-5084-635a-2752-d119c7640b2a reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=64978202-6ca0-df9b-7af8-4f3ff010b47d reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=0839351f-0605-1086-db26-a65779e5b111 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=b50e56fa-91e1-8e3c-754d-49802c364a93 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.570+0200 [INFO ] client.gc: garbage collecting allocation: alloc_id=2748dd53-e986-dbd1-0e20-ed21c847c418 reason="forced collection"
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.746+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=6854d42b-a643-0df8-5875-41f8f60537f2
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.802+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=abc490f7-b0d5-f4f0-d673-d73489c029f9
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.846+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=3abfcd2d-49d6-4989-48a3-1432e788f845
Apr 25 14:53:01 innoabis-71 nomad[3314]: 2019-04-25T14:53:01.972+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=8150fc43-79af-034b-cf0c-e14c2a800bfc
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.296+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=b89eaeab-7250-040b-3881-970baacb685e
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.296+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=c5d45bea-cc68-ae73-8319-b8960f2d90b8
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.297+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=f897d1f4-c260-bdb9-b698-8070397ee9b8
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.318+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=bfad63e1-2a86-f24f-a1a6-b11533f7074f
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.378+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=7797f96b-4ebc-3568-0f82-49a426bc98ec
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.632+0200 [WARN ] client.alloc_runner.runner_hook.alloc_health_watcher: error looking up Consul registrations for allocation: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 error="Get http://127.0.0.1:8500/v1/agent/service
Apr 25 14:53:02 innoabis-71 nomad[3314]: 2019-04-25T14:53:02.870+0200 [WARN ] client.alloc_runner.runner_hook.alloc_health_watcher: error looking up Consul registrations for allocation: alloc_id=337e2138-5084-635a-2752-d119c7640b2a error="Get http://127.0.0.1:8500/v1/agent/service
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.345+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.345+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.387+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webafis/hitlist_worker && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.454+0200 [WARN ] client.alloc_runner.runner_hook.alloc_health_watcher: error looking up Consul registrations for allocation: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 error="Get http://127.0.0.1:8500/v1/agent/service
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.585+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/db2a4831-99
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.585+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver @module=logmon path=/var/lib/nomad/alloc/db2a4831-99
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.590+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/db2a
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.590+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Websocket-Server @module=logmon path=/var/lib/nomad/alloc/db2a
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.700+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.700+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-MQ-AFIS-Worker @module=logmon path=/var/lib/nomad/alloc/db2a48
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.842+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/db2a4831-
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.842+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job @module=logmon path=/var/lib/nomad/alloc/db2a4831-
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.879+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_webserver.sh]}"
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.937+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_worker_mq.sh --
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.964+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_websocket_serve
Apr 25 14:53:03 innoabis-71 nomad[3314]: 2019-04-25T14:53:03.971+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /opt/webserver && exec /bin/bash ./bin/production_run_delayed_job.sh]
Apr 25 14:53:04 innoabis-71 nomad[3314]: 2019-04-25T14:53:04.227+0200 [WARN ] client.alloc_runner.runner_hook.alloc_health_watcher: error looking up Consul registrations for allocation: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c error="Get http://127.0.0.1:8500/v1/agent/service
Apr 25 14:53:06 innoabis-71 nomad[3314]: 2019-04-25T14:53:06.401+0200 [INFO ] client.fingerprint_mgr.consul: consul agent is available
Apr 25 14:53:06 innoabis-71 nomad[3314]: 2019-04-25T14:53:06.423+0200 [INFO ] consul.sync: successfully updated services in Consul
Apr 25 14:53:07 innoabis-71 nomad[3314]: 2019-04-25T14:53:07.172+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-Delayed-Job error="rpc error: c
Apr 25 14:53:07 innoabis-71 nomad[3314]: 2019-04-25T14:53:07.812+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Webserver error="read |0: file already closed"
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.025+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 task=IDispatcher reason="Restart within policy" delay=18.535830295s
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.422+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae driver=raw_exec task_name=Message-Processor error="rpc error: cod
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.423+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae driver=raw_exec task_name=Message-Processor error="read |0: file already closed"
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.454+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.457+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=ddc91527-a1ce-0186-e87e-b5f0366b9fae task=Message-Processor error="read |0: file already closed"
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.516+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=337e2138-5084-635a-2752-d119c7640b2a driver=raw_exec task_name=Fast-Service error="rpc error: code = U
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.620+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c driver=raw_exec task_name=Message-Processor error="rpc error: cod
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.626+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c driver=raw_exec task_name=Message-Processor error="read |0: file already closed"
Apr 25 14:53:08 innoabis-71 nomad[3314]: 2019-04-25T14:53:08.705+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=1b9bfd2b-5ed1-cea5-23fc-0535519b416c task=Message-Processor error="read |0: file already closed"
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.506+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-83
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.506+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service path=/var/lib/nomad/alloc/62233708-1df5-8387-7433-ebf2584
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.645+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5100dd6d-79bf-3c77-5b69-24993357a9ee task=activemq515 @module=logmon path=/var/lib/nomad/alloc/5100dd6d-79bf-3c7
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.645+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=5100dd6d-79bf-3c77-5b69-24993357a9ee task=activemq515 @module=logmon path=/var/lib/nomad/alloc/5100dd6d-79bf-3c7
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.675+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=ad4ddd49-783a-7129-4a61-343c087009e7 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/ad4ddd49-783a
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.675+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=ad4ddd49-783a-7129-4a61-343c087009e7 task=ABIS-Monitoring @module=logmon path=/var/lib/nomad/alloc/ad4ddd49-783a
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.689+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 26167]}"
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.745+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/external-activemq515/bin/production_run_activemq.sh]}"
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.855+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[/opt/abis-monitoring/bin/abis-monitoring.sh]}"
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.967+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e6
Apr 25 14:53:09 innoabis-71 nomad[3314]: 2019-04-25T14:53:09.967+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e6
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.052+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.262+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b driver=raw_exec task_name=activemq515 error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.263+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b driver=raw_exec task_name=activemq515 error="rpc error: code = Un
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.295+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=2232a445-10cc-3ae6-3056-92a9016f3c1b task=activemq515 error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.331+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: failed to read from log fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job @module=logmon error="read /var/lib/innovatric
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.331+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: failed to read from log fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job @module=logmon error="read /var/lib/innovatric
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.342+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Delayed-Job error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.603+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Webserver error="rpc error: cod
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.604+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Webserver error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.624+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Webserver error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.671+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-MQ-AFIS-Worker error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.672+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-MQ-AFIS-Worker error="rpc error
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.704+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-MQ-AFIS-Worker error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.814+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Delayed-Job error="rpc error: c
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.816+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Delayed-Job error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.817+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Websocket-Server error="read |0: file already closed"
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.817+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Websocket-Server error="rpc err
Apr 25 14:53:10 innoabis-71 nomad[3314]: 2019-04-25T14:53:10.984+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Websocket-Server error="read |0: file already closed"
Apr 25 14:53:11 innoabis-71 nomad[3314]: 2019-04-25T14:53:11.590+0200 [INFO ] client: node registration complete
Apr 25 14:53:12 innoabis-71 nomad[3314]: 2019-04-25T14:53:12.005+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 25 14:53:12 innoabis-71 nomad[3314]: 2019-04-25T14:53:12.005+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher reason="Restart within policy" delay=16.60166305s
Apr 25 14:53:12 innoabis-71 nomad[3314]: 2019-04-25T14:53:12.181+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-Hitlist-Worker error="rpc error
Apr 25 14:53:12 innoabis-71 nomad[3314]: 2019-04-25T14:53:12.190+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea driver=raw_exec task_name=WebAFIS-Hitlist-Worker error="read |0: file already closed"
Apr 25 14:53:12 innoabis-71 nomad[3314]: 2019-04-25T14:53:12.248+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.256+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: failed to read from log fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker error="read /var/lib/nomad/allo
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.256+0200 [WARN ] client.alloc_runner.task_runner.task_hook.logmon.nomad: failed to read from log fifo: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker error="read /var/lib/nomad/allo
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.271+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=db2a4831-9953-cfa6-3b9f-ac4a80470fea task=WebAFIS-Hitlist-Worker error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.341+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-MQ-AFIS-Worker&type=stdout error="failed to stream "alloc/lo
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.341+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 driver=raw_exec task_name=INode error="rpc error: code = Unavaila
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.344+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 driver=raw_exec task_name=INode error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.345+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Webserver&type=stdout error="failed to stream "alloc/logs/We
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.369+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Hitlist-Worker&type=stdout error="failed to stream "alloc/lo
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.403+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 task=INode error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.476+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 task=IDispatcher error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.855+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Delayed-Job&type=stdout error="failed to list entries: open
Apr 25 14:53:15 innoabis-71 nomad[3314]: 2019-04-25T14:53:15.555+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Hitlist-Worker error="rpc error
Apr 25 14:53:15 innoabis-71 nomad[3314]: 2019-04-25T14:53:15.619+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker error="read |0: file already closed"
Apr 25 14:53:24 innoabis-71 nomad[3314]: 2019-04-25T14:53:24.275+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:24 innoabis-71 nomad[3314]: 2019-04-25T14:53:24.755+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:28 innoabis-71 nomad[3314]: 2019-04-25T14:53:28.631+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher path=/var/lib/nomad/alloc/14d9d55e-0760-9e64-385e-eef6fbfb
Apr 25 14:53:28 innoabis-71 nomad[3314]: 2019-04-25T14:53:28.631+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e6
Apr 25 14:53:28 innoabis-71 nomad[3314]: 2019-04-25T14:53:28.632+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 25 14:53:30 innoabis-71 nomad[3314]: 2019-04-25T14:53:30.774+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.163+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.485+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.486+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.486+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher reason="Restart within policy" delay=18.00636782s
Apr 25 14:53:36 innoabis-71 nomad[3314]: 2019-04-25T14:53:36.643+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:38 innoabis-71 nomad[3314]: 2019-04-25T14:53:38.235+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:39 innoabis-71 nomad[3314]: 2019-04-25T14:53:39.876+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:42 innoabis-71 nomad[3314]: 2019-04-25T14:53:42.336+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:44 innoabis-71 nomad[3314]: 2019-04-25T14:53:44.148+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:46 innoabis-71 nomad[3314]: 2019-04-25T14:53:46.284+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e driver=raw_exec task_name=Fast-Service error="rpc error: code = U
Apr 25 14:53:46 innoabis-71 nomad[3314]: 2019-04-25T14:53:46.285+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e driver=raw_exec task_name=Fast-Service error="read |0: file already closed"
Apr 25 14:53:46 innoabis-71 nomad[3314]: 2019-04-25T14:53:46.285+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service reason="Restart within policy" delay=16.685710739s
Apr 25 14:53:49 innoabis-71 nomad[3314]: 2019-04-25T14:53:49.539+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e6
Apr 25 14:53:49 innoabis-71 nomad[3314]: 2019-04-25T14:53:49.539+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e6
Apr 25 14:53:49 innoabis-71 nomad[3314]: 2019-04-25T14:53:49.540+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.861+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="rpc error: code = Un
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.861+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.861+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.903+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323
Apr 25 14:54:02 innoabis-71 nomad[3314]: 2019-04-25T14:54:02.995+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-83
Apr 25 14:54:02 innoabis-71 nomad[3314]: 2019-04-25T14:54:02.995+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-83
Apr 25 14:54:02 innoabis-71 nomad[3314]: 2019-04-25T14:54:02.997+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 26167]}"
Apr 25 14:54:18 innoabis-71 nomad[3314]: 2019-04-25T14:54:18.516+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e driver=raw_exec task_name=Fast-Service error="rpc error: code = U
Apr 25 14:54:18 innoabis-71 nomad[3314]: 2019-04-25T14:54:18.521+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service reason="Restart within policy" delay=18.655639936s
Apr 25 14:54:37 innoabis-71 nomad[3314]: 2019-04-25T14:54:37.191+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-83
Apr 25 14:54:37 innoabis-71 nomad[3314]: 2019-04-25T14:54:37.191+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-83
Apr 25 14:54:37 innoabis-71 nomad[3314]: 2019-04-25T14:54:37.193+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 26167]}"
lines 317-358/358 (END)
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.341+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 driver=raw_exec task_name=INode error="rpc error: code = Unavailable desc = transport is closing"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.344+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 driver=raw_exec task_name=INode error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.345+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Webserver&type=stdout error="failed to stream "alloc/logs/WebAFIS-Webserver.stdout.0": failed to
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.369+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Hitlist-Worker&type=stdout error="failed to stream "alloc/logs/WebAFIS-Hitlist-Worker.stdout.0":
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.403+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e07196e7-5dea-ca85-63a4-97dfb3f25554 task=INode error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.476+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=5b8a48f3-b817-0b0e-9223-c53096437a26 task=IDispatcher error="read |0: file already closed"
Apr 25 14:53:14 innoabis-71 nomad[3314]: 2019-04-25T14:53:14.855+0200 [ERROR] http: request failed: method=GET path=/v1/client/fs/logs/db2a4831-9953-cfa6-3b9f-ac4a80470fea?follow=true&offset=50000&origin=end&task=WebAFIS-Delayed-Job&type=stdout error="failed to list entries: open /var/lib/nomad/alloc/db2
Apr 25 14:53:15 innoabis-71 nomad[3314]: 2019-04-25T14:53:15.555+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 driver=raw_exec task_name=WebAFIS-Hitlist-Worker error="rpc error: code = Unavailable desc = transpor
Apr 25 14:53:15 innoabis-71 nomad[3314]: 2019-04-25T14:53:15.619+0200 [ERROR] client.alloc_runner.task_runner.task_hook.logmon.nomad: reading plugin stderr: alloc_id=e0d67b3c-1aa5-967f-bc24-d42bf64744a5 task=WebAFIS-Hitlist-Worker error="read |0: file already closed"
Apr 25 14:53:24 innoabis-71 nomad[3314]: 2019-04-25T14:53:24.275+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:24 innoabis-71 nomad[3314]: 2019-04-25T14:53:24.755+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:28 innoabis-71 nomad[3314]: 2019-04-25T14:53:28.631+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher path=/var/lib/nomad/alloc/14d9d55e-0760-9e64-385e-eef6fbfb1323/alloc/logs/.IDispatcher.stdout.
Apr 25 14:53:28 innoabis-71 nomad[3314]: 2019-04-25T14:53:28.631+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e64-385e-eef6fbfb1323/alloc/logs/.IDis
Apr 25 14:53:28 innoabis-71 nomad[3314]: 2019-04-25T14:53:28.632+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 25 14:53:30 innoabis-71 nomad[3314]: 2019-04-25T14:53:30.774+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.163+0200 [ERROR] http: request failed: method=GET path=/v1/namespace/default error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.485+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.486+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 25 14:53:31 innoabis-71 nomad[3314]: 2019-04-25T14:53:31.486+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher reason="Restart within policy" delay=18.00636782s
Apr 25 14:53:36 innoabis-71 nomad[3314]: 2019-04-25T14:53:36.643+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:38 innoabis-71 nomad[3314]: 2019-04-25T14:53:38.235+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:39 innoabis-71 nomad[3314]: 2019-04-25T14:53:39.876+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:42 innoabis-71 nomad[3314]: 2019-04-25T14:53:42.336+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:44 innoabis-71 nomad[3314]: 2019-04-25T14:53:44.148+0200 [ERROR] http: request failed: method=GET path=/v1/namespaces error="Nomad Enterprise only endpoint" code=501
Apr 25 14:53:46 innoabis-71 nomad[3314]: 2019-04-25T14:53:46.284+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e driver=raw_exec task_name=Fast-Service error="rpc error: code = Unavailable desc = transport is closi
Apr 25 14:53:46 innoabis-71 nomad[3314]: 2019-04-25T14:53:46.285+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e driver=raw_exec task_name=Fast-Service error="read |0: file already closed"
Apr 25 14:53:46 innoabis-71 nomad[3314]: 2019-04-25T14:53:46.285+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service reason="Restart within policy" delay=16.685710739s
Apr 25 14:53:49 innoabis-71 nomad[3314]: 2019-04-25T14:53:49.539+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e64-385e-eef6fbfb1323/alloc/logs/.IDis
Apr 25 14:53:49 innoabis-71 nomad[3314]: 2019-04-25T14:53:49.539+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher @module=logmon path=/var/lib/nomad/alloc/14d9d55e-0760-9e64-385e-eef6fbfb1323/alloc/logs/.IDis
Apr 25 14:53:49 innoabis-71 nomad[3314]: 2019-04-25T14:53:49.540+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c cd /srv/apps/eafis/natives && exec ./IDispatcher -c dispatcher-config.xml -d 1]}"
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.861+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="rpc error: code = Unavailable desc = transport is closin
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.861+0200 [ERROR] client.driver_mgr.raw_exec.executor.nomad: reading plugin stderr: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 driver=raw_exec task_name=IDispatcher error="read |0: file already closed"
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.861+0200 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323 task=IDispatcher reason="Exceeded allowed attempts 2 in interval 30m0s and mode is "fail""
Apr 25 14:53:50 innoabis-71 nomad[3314]: 2019-04-25T14:53:50.903+0200 [INFO ] client.gc: marking allocation for GC: alloc_id=14d9d55e-0760-9e64-385e-eef6fbfb1323
Apr 25 14:54:02 innoabis-71 nomad[3314]: 2019-04-25T14:54:02.995+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-8387-7433-ebf2584f7c4e/alloc/logs/.Fas
Apr 25 14:54:02 innoabis-71 nomad[3314]: 2019-04-25T14:54:02.995+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-8387-7433-ebf2584f7c4e/alloc/logs/.Fas
Apr 25 14:54:02 innoabis-71 nomad[3314]: 2019-04-25T14:54:02.997+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 26167]}"
Apr 25 14:54:18 innoabis-71 nomad[3314]: 2019-04-25T14:54:18.516+0200 [ERROR] client.driver_mgr.raw_exec: error receiving stream from Stats executor RPC, closing stream: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e driver=raw_exec task_name=Fast-Service error="rpc error: code = Unavailable desc = transport is closi
Apr 25 14:54:18 innoabis-71 nomad[3314]: 2019-04-25T14:54:18.521+0200 [INFO ] client.alloc_runner.task_runner: restarting task: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service reason="Restart within policy" delay=18.655639936s
Apr 25 14:54:37 innoabis-71 nomad[3314]: 2019-04-25T14:54:37.191+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-8387-7433-ebf2584f7c4e/alloc/logs/.Fas
Apr 25 14:54:37 innoabis-71 nomad[3314]: 2019-04-25T14:54:37.191+0200 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=62233708-1df5-8387-7433-ebf2584f7c4e task=Fast-Service @module=logmon path=/var/lib/nomad/alloc/62233708-1df5-8387-7433-ebf2584f7c4e/alloc/logs/.Fas
Apr 25 14:54:37 innoabis-71 nomad[3314]: 2019-04-25T14:54:37.193+0200 [INFO ] client.driver_mgr.raw_exec: starting task: driver=raw_exec driver_cfg="{Command:/bin/bash Args:[-c exec /bin/bash /opt/eafis/eafis-cluster/eafis-fast-service.sh -adminPort 26167]}"

To: nomad-oss-debug@hashicorp.com

Subject: GH-1234: Errors garbage collecting allocs

Emails sent to that address are readable by all HashiCorp employees but are not publicly visible.

Nomad Server logs (if appropriate)
@notnoop notnoop self-assigned this Apr 30, 2019
@notnoop
Copy link
Contributor

notnoop commented Apr 30, 2019

Thanks for reporting this bug! I'll dig into this more this week and report findings.

@prologic
Copy link

prologic commented May 2, 2019 •
edited

I can reproduce this as well by simply rebooting a Nomad node.

Task hook failed  logmon: Reattachment process not found

Version:

Nomad v0.9.0-dev (7feee07d8f353f4bc83a19d25bb6906ed98edb0c+CHANGES)

@endocrimes
Copy link
Contributor

@prologic The logmon reattachment bug was fixed as part of 0.9.1 in #5577

@prologic
Copy link

prologic commented May 2, 2019

@prologic The logmon reattachment bug was fixed as part of 0.9.1 in #5577

Thank you! We will upgrade our Nomad binary and confirm and comment back here. /cc @johnzhanghua

@prologic
Copy link

prologic commented May 7, 2019

Confirm upgrading to 0.9.1 resolves this and I no longer see Task hook failed logmon: Reattachment process not found against all tasks after a reboot.

@johnzhanghua
Copy link

johnzhanghua commented May 7, 2019 via email

@preetapan
Copy link
Contributor

CLosing per comment from @prologic

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 23, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees

@notnoop notnoop

Labels
stage/needs-investigation theme/client type/bug
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
6 participants
@notnoop @preetapan @prologic @endocrimes @jozef-slezak @johnzhanghua