Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Restarting agent reruns successfully completed allocations #5945

Closed
ilyaiqoqo opened this issue Jul 9, 2019 · 3 comments
Closed

Restarting agent reruns successfully completed allocations #5945

ilyaiqoqo opened this issue Jul 9, 2019 · 3 comments

Comments

@ilyaiqoqo
Copy link
Contributor

ilyaiqoqo commented Jul 9, 2019 •
edited

Nomad version

Nomad v0.9.3 (c5e8b66)

Operating system and Environment details

Darwin ilyas-MacBook-Pro.local 18.2.0 Darwin Kernel Version 18.2.0: Fri Oct 5 19:41:49 PDT 2018; root:xnu-4903.221.2~2/RELEASE_X86_64 x86_64

Issue

Dispatched task allocations are rerun on agent restart although successfully completed

Reproduction steps

  1. Dispatch nomad job
  2. Wait for the job to finish successfully
  3. stop agent
  4. start agent
  5. successfully completed allocations are rerun

The expected behavior is not to run successfully completed tasks

Job file (if appropriate)

{
  "Stop": false,
  "Region": "us-west-2",
  "Namespace": "default",
  "ID": "5d250666572370000b79513a/dispatch-1562707559-dd1c74e5",
  "ParentID": "5d250666572370000b79513a",
  "Name": "5d250666572370000b79513a/dispatch-1562707559-dd1c74e5",
  "Type": "batch",
  "Priority": 50,
  "AllAtOnce": false,
  "Datacenters": [
    "us-west-2a",
    "us-west-2b",
    "us-west-2c",
    "global"
  ],
  "Constraints": [
    {
      "LTarget": "${meta.ClusterID}",
      "RTarget": "000000000000000000000000",
      "Operand": "="
    }
  ],
  "Affinities": null,
  "Spreads": null,
  "TaskGroups": [
    {
      "Name": "IqoqoGroup",
      "Count": 1,
      "Update": null,
      "Migrate": null,
      "Constraints": null,
      "RestartPolicy": {
        "Attempts": 3,
        "Interval": 86400000000000,
        "Delay": 15000000000,
        "Mode": "fail"
      },
      "Tasks": [
        {
          "Name": "IqoqoTask",
          "Driver": "docker",
          "User": "",
          "Config": {
            "args": [
              "${NOMAD_META_ARGUMENT_0}",
              "${NOMAD_META_ARGUMENT_1}"
            ],
            "force_pull": true,
            "work_dir": "/local",
            "image": "iqoqo/python-runtime:latest",
            "entrypoint": [
              "python"
            ]
          },
          "Env": null,
          "Services": null,
          "Vault": null,
          "Templates": null,
          "Constraints": null,
          "Affinities": null,
          "Resources": {
            "CPU": 6000,
            "MemoryMB": 3700,
            "DiskMB": 0,
            "IOPS": 0,
            "Networks": null,
            "Devices": null
          },
          "DispatchPayload": null,
          "Meta": null,
          "KillTimeout": 5000000000,
          "LogConfig": {
            "MaxFiles": 10,
            "MaxFileSizeMB": 10
          },
          "Artifacts": [
            {
              "GetterSource": "${NOMAD_META_ARTIFACT_SOURCE_0}",
              "GetterOptions": null,
              "GetterMode": "any",
              "RelativeDest": "local/"
            },
            {
              "GetterSource": "${NOMAD_META_ARTIFACT_SOURCE_1}",
              "GetterOptions": null,
              "GetterMode": "any",
              "RelativeDest": "local/"
            }
          ],
          "Leader": false,
          "ShutdownDelay": 0,
          "KillSignal": ""
        }
      ],
      "EphemeralDisk": {
        "Sticky": false,
        "SizeMB": 300,
        "Migrate": false
      },
      "Meta": null,
      "ReschedulePolicy": {
        "Attempts": 1,
        "Interval": 86400000000000,
        "Delay": 5000000000,
        "DelayFunction": "constant",
        "MaxDelay": 0,
        "Unlimited": false
      },
      "Affinities": null,
      "Spreads": null
    }
  ],
  "Update": {
    "Stagger": 0,
    "MaxParallel": 0,
    "HealthCheck": "",
    "MinHealthyTime": 0,
    "HealthyDeadline": 0,
    "ProgressDeadline": 0,
    "AutoRevert": false,
    "AutoPromote": false,
    "Canary": 0
  },
  "Periodic": null,
  "ParameterizedJob": {
    "Payload": "optional",
    "MetaRequired": [
      "TASK_ID",
      "TASK_TOKEN",
      "TASK_ARGS",
      "RESULT_DESTINATION",
      "ARTIFACT_SOURCE_0",
      "ARTIFACT_SOURCE_1",
      "ARGUMENT_0",
      "ARGUMENT_1"
    ],
    "MetaOptional": null
  },
  "Dispatched": true,
  "Payload": null,
  "Meta": {
    "ClusterID": "000000000000000000000000",
    "TASK_ARGS": "[\"helloworld.py\",\"82\"]",
    "ARGUMENT_1": "82",
    "ARGUMENT_0": "helloworld.py",
    "ClusterInstanceType": "s",
    "TASK_ID": "5d250666572370000b795187",
    "RESULT_DESTINATION": "REDACTED",
    "ClusterType": "aws",
    "ARTIFACT_SOURCE_1": "REDACTED",
    "TASK_TOKEN": "REDACTED",
    "ARTIFACT_SOURCE_0": "REDACTED"
  },
  "VaultToken": "",
  "Status": "dead",
  "StatusDescription": "",
  "Stable": false,
  "Version": 0,
  "SubmitTime": 1562707559127288800,
  "CreateIndex": 830373,
  "ModifyIndex": 831879,
  "JobModifyIndex": 830373
}

Nomad Client logs (if appropriate)

sudo ./nomad agent -client -servers "nomadrpc.dev.iqoqo.co" -meta "ClusterID=000000000000000000000000" -meta "ClusterInstanceType=s" -data-dir /tmp/iqoqo-agent -dc "global" -region "us-west-2" -config /tmp/iqoqo-agent/config
==> Loaded configuration from /tmp/iqoqo-agent/config/config.hcl, /tmp/iqoqo-agent/config/tls.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 10.0.0.4:4646
            Bind Addrs: HTTP: 0.0.0.0:4646
                Client: true
             Log Level: INFO
                Region: us-west-2 (DC: global)
                Server: false
               Version: 0.9.3

==> Nomad agent started! Log data will stream in below:

    2019-07-10T01:26:13.285+0300 [WARN ] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/tmp/iqoqo-agent/plugins
    2019-07-10T01:26:13.286+0300 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2019-07-10T01:26:13.286+0300 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2019-07-10T01:26:13.286+0300 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2019-07-10T01:26:13.286+0300 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2019-07-10T01:26:13.286+0300 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2019-07-10T01:26:13.294+0300 [INFO ] client: using state directory: state_dir=/tmp/iqoqo-agent/client
    2019-07-10T01:26:13.295+0300 [INFO ] client: using alloc directory: alloc_dir=/tmp/iqoqo-agent/alloc
    2019-07-10T01:26:17.414+0300 [INFO ] client.plugin: starting plugin manager: plugin-type=driver
    2019-07-10T01:26:17.414+0300 [INFO ] client.plugin: starting plugin manager: plugin-type=device
    2019-07-10T01:26:18.378+0300 [INFO ] client: started client: node_id=ec241d5d-e1ff-3b5c-fd4f-510269a65290
    2019-07-10T01:26:18.608+0300 [INFO ] client: node registration complete
    2019-07-10T01:26:27.400+0300 [INFO ] client: node registration complete
    2019-07-10T01:26:50.143+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/8775342c-6060-f733-b356-981820e71501/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:26:50.143+0300
    2019-07-10T01:26:50.143+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/50120b73-c417-c782-4b85-651128935a58/alloc/logs/.IqoqoTask.stdout.fifo @module=logmon timestamp=2019-07-10T01:26:50.143+0300
    2019-07-10T01:26:50.143+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/0cef5646-5694-66c9-ef81-7f8a2485c478/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:26:50.143+0300
    2019-07-10T01:26:50.143+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/8775342c-6060-f733-b356-981820e71501/alloc/logs/.IqoqoTask.stderr.fifo @module=logmon timestamp=2019-07-10T01:26:50.143+0300
    2019-07-10T01:26:50.143+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/0cef5646-5694-66c9-ef81-7f8a2485c478/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:26:50.143+0300
    2019-07-10T01:26:50.143+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/50120b73-c417-c782-4b85-651128935a58/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:26:50.143+0300
    2019-07-10T01:26:56.379+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=1908e9408d8dc648ac95c4e56d176dd1e5da9fb6d455c549983ef6aa40adb9f3
    2019-07-10T01:26:56.411+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=0ecb1dd9fc15cc6272d0069602c1c7a4273e572cd565bf4c5a614a5336515e87
    2019-07-10T01:26:56.424+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=09802cf7e3dd39e22aa9ada8e111d3bda7fbc2c315ec52f133b8626012b08401
    2019-07-10T01:26:57.232+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=0ecb1dd9fc15cc6272d0069602c1c7a4273e572cd565bf4c5a614a5336515e87
    2019-07-10T01:26:57.532+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=09802cf7e3dd39e22aa9ada8e111d3bda7fbc2c315ec52f133b8626012b08401
    2019-07-10T01:26:57.608+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=1908e9408d8dc648ac95c4e56d176dd1e5da9fb6d455c549983ef6aa40adb9f3
    2019-07-10T01:26:58.071+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:26:58.072+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=8775342c-6060-f733-b356-981820e71501
    2019-07-10T01:26:58.492+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:26:58.492+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478
    2019-07-10T01:26:58.703+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:26:58.704+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=50120b73-c417-c782-4b85-651128935a58
    2019-07-10T01:26:58.969+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6/alloc/logs/.IqoqoTask.stdout.fifo @module=logmon timestamp=2019-07-10T01:26:58.969+0300
    2019-07-10T01:26:58.969+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:26:58.969+0300
    2019-07-10T01:26:59.432+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/53530227-5dd1-93a4-28ae-70575bc33162/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:26:59.431+0300
    2019-07-10T01:26:59.432+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/53530227-5dd1-93a4-28ae-70575bc33162/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:26:59.432+0300
    2019-07-10T01:26:59.883+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask path=/tmp/iqoqo-agent/alloc/9ffb644d-cade-7deb-c2c2-ef5b1961ccdb/alloc/logs/.IqoqoTask.stdout.fifo @module=logmon timestamp=2019-07-10T01:26:59.882+0300
    2019-07-10T01:26:59.883+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9ffb644d-cade-7deb-c2c2-ef5b1961ccdb/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:26:59.883+0300
    2019-07-10T01:27:05.029+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=58f12a5031c1742405d0412d99550120ce8a2c0227e37e92b52c1c9672ccf6b2
    2019-07-10T01:27:05.031+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=44ba866a6a24bfb99daead411527ed65bbc254c7a721db6ce338897653e6304c
    2019-07-10T01:27:05.043+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=9cd50ea7b67fcb7f788d82b5878015c09e4c1756820e8f075b159bea4f09f727
    2019-07-10T01:27:05.874+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=44ba866a6a24bfb99daead411527ed65bbc254c7a721db6ce338897653e6304c
    2019-07-10T01:27:06.123+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=58f12a5031c1742405d0412d99550120ce8a2c0227e37e92b52c1c9672ccf6b2
    2019-07-10T01:27:06.181+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=9cd50ea7b67fcb7f788d82b5878015c09e4c1756820e8f075b159bea4f09f727
    2019-07-10T01:27:07.037+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:07.037+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162
    2019-07-10T01:27:07.156+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:07.157+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb
    2019-07-10T01:27:07.281+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:07.281+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6
    2019-07-10T01:27:07.983+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/36688829-cf14-3c3b-0fce-f06853bc4b45/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:27:07.983+0300
    2019-07-10T01:27:07.983+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9f3cd741-b145-6596-d440-2a2cee8ef8c9/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:27:07.983+0300
    2019-07-10T01:27:07.983+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9f3cd741-b145-6596-d440-2a2cee8ef8c9/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:27:07.983+0300
    2019-07-10T01:27:07.984+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/36688829-cf14-3c3b-0fce-f06853bc4b45/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:27:07.983+0300
    2019-07-10T01:27:08.678+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/79d2ff9a-3825-be3a-385b-137a2fa40186/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:27:08.678+0300
    2019-07-10T01:27:08.679+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/79d2ff9a-3825-be3a-385b-137a2fa40186/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:27:08.679+0300
    2019-07-10T01:27:14.002+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=9d8018233cf2b6517e4bf5c85f94ae6055fc7c4f4fffef53da6dbf7df47f21aa
    2019-07-10T01:27:14.002+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=0ba707f72f620add189a09dd9444ce5abaa5676e3408d4a6140dd39c6e5895f7
    2019-07-10T01:27:14.004+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=1c1a499b071ce3a8bea4e85bac36f5238e3781953e8343c231bbd5b0347dfa0b
    2019-07-10T01:27:14.951+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=0ba707f72f620add189a09dd9444ce5abaa5676e3408d4a6140dd39c6e5895f7
    2019-07-10T01:27:15.081+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=9d8018233cf2b6517e4bf5c85f94ae6055fc7c4f4fffef53da6dbf7df47f21aa
    2019-07-10T01:27:15.219+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=1c1a499b071ce3a8bea4e85bac36f5238e3781953e8343c231bbd5b0347dfa0b
    2019-07-10T01:27:15.764+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:15.765+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186
    2019-07-10T01:27:16.062+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:16.063+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9
    2019-07-10T01:27:16.174+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:16.175+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45
    2019-07-10T01:27:17.151+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/793e0894-26a9-d9b4-56f7-755e097d226a/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:27:17.151+0300
    2019-07-10T01:27:17.152+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/793e0894-26a9-d9b4-56f7-755e097d226a/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:27:17.152+0300
    2019-07-10T01:27:17.154+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/95a901c2-b206-e7e3-16d4-3fd261a753bb/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:27:17.154+0300
    2019-07-10T01:27:17.156+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/95a901c2-b206-e7e3-16d4-3fd261a753bb/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:27:17.155+0300
    2019-07-10T01:27:17.157+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/31f62ca4-5757-cfc0-a808-39ac99535eea/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:27:17.157+0300
    2019-07-10T01:27:17.157+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask path=/tmp/iqoqo-agent/alloc/31f62ca4-5757-cfc0-a808-39ac99535eea/alloc/logs/.IqoqoTask.stderr.fifo @module=logmon timestamp=2019-07-10T01:27:17.157+0300
    2019-07-10T01:27:23.014+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=6a5bb016dd8c424e30ac0fbdcbac565164818254081567e6cbbeaad41cba1042
    2019-07-10T01:27:23.015+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=182c848fccc50631faeb276321ae01e5657ea06a904ea72d1f92f58824e78b45
    2019-07-10T01:27:23.017+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=6217cf7aeedba64eadf07ca821142484f03ed5088f2c4d83a5fa9ef83dedc202
    2019-07-10T01:27:23.791+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=182c848fccc50631faeb276321ae01e5657ea06a904ea72d1f92f58824e78b45
    2019-07-10T01:27:23.948+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=6a5bb016dd8c424e30ac0fbdcbac565164818254081567e6cbbeaad41cba1042
    2019-07-10T01:27:24.093+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=6217cf7aeedba64eadf07ca821142484f03ed5088f2c4d83a5fa9ef83dedc202
    2019-07-10T01:27:24.751+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:24.751+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb
    2019-07-10T01:27:24.806+0300 [ERROR] client.driver_mgr.docker.docker_logger.nomad: reading plugin stderr: driver=docker error="read |0: file already closed"
    2019-07-10T01:27:24.821+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:24.822+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a
    2019-07-10T01:27:24.968+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:27:24.969+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea
^C==> Caught signal: interrupt
    2019-07-10T01:28:30.659+0300 [INFO ] agent: requesting shutdown
    2019-07-10T01:28:30.659+0300 [INFO ] client: shutting down
    2019-07-10T01:28:30.673+0300 [INFO ] client.plugin: shutting down plugin manager: plugin-type=device
    2019-07-10T01:28:30.673+0300 [INFO ] client.plugin: plugin manager finished: plugin-type=device
    2019-07-10T01:28:30.673+0300 [INFO ] client.plugin: shutting down plugin manager: plugin-type=driver
    2019-07-10T01:28:30.675+0300 [INFO ] client.plugin: plugin manager finished: plugin-type=driver
    2019-07-10T01:28:30.680+0300 [INFO ] agent: shutdown complete

starting nomad agent again

sudo ./nomad agent -client -servers "nomadrpc.dev.iqoqo.co" -meta "ClusterID=000000000000000000000000" -meta "ClusterInstanceType=s" -data-dir /tmp/iqoqo-agent -dc "global" -region "us-west-2" -config /tmp/iqoqo-agent/config
==> Loaded configuration from /tmp/iqoqo-agent/config/config.hcl, /tmp/iqoqo-agent/config/tls.hcl
==> Starting Nomad agent...
==> Nomad agent configuration:

       Advertise Addrs: HTTP: 10.0.0.4:4646
            Bind Addrs: HTTP: 0.0.0.0:4646
                Client: true
             Log Level: INFO
                Region: us-west-2 (DC: global)
                Server: false
               Version: 0.9.3

==> Nomad agent started! Log data will stream in below:

    2019-07-10T01:28:33.530+0300 [WARN ] agent.plugin_loader: skipping external plugins since plugin_dir doesn't exist: plugin_dir=/tmp/iqoqo-agent/plugins
    2019-07-10T01:28:33.531+0300 [INFO ] agent: detected plugin: name=docker type=driver plugin_version=0.1.0
    2019-07-10T01:28:33.531+0300 [INFO ] agent: detected plugin: name=raw_exec type=driver plugin_version=0.1.0
    2019-07-10T01:28:33.531+0300 [INFO ] agent: detected plugin: name=exec type=driver plugin_version=0.1.0
    2019-07-10T01:28:33.531+0300 [INFO ] agent: detected plugin: name=qemu type=driver plugin_version=0.1.0
    2019-07-10T01:28:33.531+0300 [INFO ] agent: detected plugin: name=java type=driver plugin_version=0.1.0
    2019-07-10T01:28:33.539+0300 [INFO ] client: using state directory: state_dir=/tmp/iqoqo-agent/client
    2019-07-10T01:28:33.539+0300 [INFO ] client: using alloc directory: alloc_dir=/tmp/iqoqo-agent/alloc
    2019-07-10T01:28:37.659+0300 [INFO ] client.plugin: starting plugin manager: plugin-type=driver
    2019-07-10T01:28:37.659+0300 [INFO ] client.plugin: starting plugin manager: plugin-type=device
    2019-07-10T01:28:38.788+0300 [INFO ] client: started client: node_id=ec241d5d-e1ff-3b5c-fd4f-510269a65290
    2019-07-10T01:28:38.789+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.790+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.790+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.796+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.797+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.799+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.799+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.800+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.801+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.801+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.803+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.803+0300 [WARN ] client.alloc_runner.task_runner.task_hook: failed to reattach to logmon process: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask error="Reattachment process not found"
    2019-07-10T01:28:38.905+0300 [INFO ] client: node registration complete
    2019-07-10T01:28:38.918+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/793e0894-26a9-d9b4-56f7-755e097d226a/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.918+0300
    2019-07-10T01:28:38.918+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask path=/tmp/iqoqo-agent/alloc/793e0894-26a9-d9b4-56f7-755e097d226a/alloc/logs/.IqoqoTask.stderr.fifo @module=logmon timestamp=2019-07-10T01:28:38.918+0300
    2019-07-10T01:28:38.922+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a
    2019-07-10T01:28:38.936+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6/alloc/logs/.IqoqoTask.stdout.fifo @module=logmon timestamp=2019-07-10T01:28:38.936+0300
    2019-07-10T01:28:38.936+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.936+0300
    2019-07-10T01:28:38.937+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/36688829-cf14-3c3b-0fce-f06853bc4b45/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.937+0300
    2019-07-10T01:28:38.938+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/36688829-cf14-3c3b-0fce-f06853bc4b45/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.937+0300
    2019-07-10T01:28:38.939+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6
    2019-07-10T01:28:38.940+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45
    2019-07-10T01:28:38.942+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/0cef5646-5694-66c9-ef81-7f8a2485c478/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.941+0300
    2019-07-10T01:28:38.942+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/0cef5646-5694-66c9-ef81-7f8a2485c478/alloc/logs/.IqoqoTask.stderr.fifo @module=logmon timestamp=2019-07-10T01:28:38.942+0300
    2019-07-10T01:28:38.943+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/79d2ff9a-3825-be3a-385b-137a2fa40186/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.943+0300
    2019-07-10T01:28:38.944+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/79d2ff9a-3825-be3a-385b-137a2fa40186/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.944+0300
    2019-07-10T01:28:38.949+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478
    2019-07-10T01:28:38.951+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186
    2019-07-10T01:28:38.981+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/95a901c2-b206-e7e3-16d4-3fd261a753bb/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.981+0300
    2019-07-10T01:28:38.981+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask path=/tmp/iqoqo-agent/alloc/95a901c2-b206-e7e3-16d4-3fd261a753bb/alloc/logs/.IqoqoTask.stderr.fifo @module=logmon timestamp=2019-07-10T01:28:38.981+0300
    2019-07-10T01:28:38.986+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb
    2019-07-10T01:28:38.992+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/53530227-5dd1-93a4-28ae-70575bc33162/alloc/logs/.IqoqoTask.stdout.fifo @module=logmon timestamp=2019-07-10T01:28:38.992+0300
    2019-07-10T01:28:38.992+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/53530227-5dd1-93a4-28ae-70575bc33162/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.992+0300
    2019-07-10T01:28:38.992+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask path=/tmp/iqoqo-agent/alloc/8775342c-6060-f733-b356-981820e71501/alloc/logs/.IqoqoTask.stdout.fifo @module=logmon timestamp=2019-07-10T01:28:38.992+0300
    2019-07-10T01:28:38.993+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/8775342c-6060-f733-b356-981820e71501/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.993+0300
    2019-07-10T01:28:38.994+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9f3cd741-b145-6596-d440-2a2cee8ef8c9/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.994+0300
    2019-07-10T01:28:38.995+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9f3cd741-b145-6596-d440-2a2cee8ef8c9/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.994+0300
    2019-07-10T01:28:38.995+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162
    2019-07-10T01:28:38.997+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/50120b73-c417-c782-4b85-651128935a58/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.997+0300
    2019-07-10T01:28:38.997+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=8775342c-6060-f733-b356-981820e71501
    2019-07-10T01:28:38.997+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/50120b73-c417-c782-4b85-651128935a58/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.997+0300
    2019-07-10T01:28:38.998+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/31f62ca4-5757-cfc0-a808-39ac99535eea/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:38.998+0300
    2019-07-10T01:28:38.998+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/31f62ca4-5757-cfc0-a808-39ac99535eea/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:38.998+0300
    2019-07-10T01:28:39.000+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9
    2019-07-10T01:28:39.000+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=50120b73-c417-c782-4b85-651128935a58
    2019-07-10T01:28:39.003+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea
    2019-07-10T01:28:39.018+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9ffb644d-cade-7deb-c2c2-ef5b1961ccdb/alloc/logs/.IqoqoTask.stdout.fifo timestamp=2019-07-10T01:28:39.018+0300
    2019-07-10T01:28:39.019+0300 [INFO ] client.alloc_runner.task_runner.task_hook.logmon.nomad: opening fifo: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask @module=logmon path=/tmp/iqoqo-agent/alloc/9ffb644d-cade-7deb-c2c2-ef5b1961ccdb/alloc/logs/.IqoqoTask.stderr.fifo timestamp=2019-07-10T01:28:39.019+0300
    2019-07-10T01:28:39.022+0300 [INFO ] client.gc: marking allocation for GC: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb
    2019-07-10T01:28:41.286+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=de699f63a91bec4dd2e08a7e3d50a71fe977ce5495d497677011d0fec18f5e52
    2019-07-10T01:28:41.286+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=0d327c9db89f6b7c48fbc396b47c4732e968f1fa6585c5021b85001eb73262a6
    2019-07-10T01:28:41.286+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=624f9ab9f0f054078ee7021af978716321a7155efc771126136704deaddf77b4
    2019-07-10T01:28:41.286+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=7b6f29026af49ba91ce0defe6d12373bf591646c80d224e07486cc1f6dcd842d
    2019-07-10T01:28:41.288+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=28f265d730560df730a7a32cbc20460a74cee7756a599ba24ac0933d8d5cd965
    2019-07-10T01:28:41.288+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=812d5259939ccaa03087d017cdcfc07baa95b5d4d45c458603e2c5e8b099d1d2
    2019-07-10T01:28:41.324+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=1943d7794fd8c845d3d2efec5bd0fe59ebc15c81774887b9201f8497ff1fb2f2
    2019-07-10T01:28:41.329+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=1c1c80c9a1f9f56a921b58f3688c647d95e95bfed0a675c89935a6fb31f07b2a
    2019-07-10T01:28:41.330+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=9d330a907ad2ba85ddf7862d68fe3563dc2e92153b513a2751af5a2aceefc7e6
    2019-07-10T01:28:41.331+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=159737cc04222f17f6c103f3e2f96fb3ee256ca22386b13486fbbc5303ad1d44
    2019-07-10T01:28:41.339+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=21547fa1ebd63af111a824afee03c5420c5a7a5386ad22ae4b3935d679226c8a
    2019-07-10T01:28:41.348+0300 [INFO ] client.driver_mgr.docker: created container: driver=docker container_id=9517d04ab05b0c8518823120401a93a888a40393d186fa1e519540456838ae99
    2019-07-10T01:28:42.608+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=159737cc04222f17f6c103f3e2f96fb3ee256ca22386b13486fbbc5303ad1d44
    2019-07-10T01:28:42.684+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=812d5259939ccaa03087d017cdcfc07baa95b5d4d45c458603e2c5e8b099d1d2
    2019-07-10T01:28:42.785+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=de699f63a91bec4dd2e08a7e3d50a71fe977ce5495d497677011d0fec18f5e52
    2019-07-10T01:28:43.036+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=1943d7794fd8c845d3d2efec5bd0fe59ebc15c81774887b9201f8497ff1fb2f2
    2019-07-10T01:28:43.249+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=1c1c80c9a1f9f56a921b58f3688c647d95e95bfed0a675c89935a6fb31f07b2a
    2019-07-10T01:28:43.493+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=9517d04ab05b0c8518823120401a93a888a40393d186fa1e519540456838ae99
    2019-07-10T01:28:43.726+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=7b6f29026af49ba91ce0defe6d12373bf591646c80d224e07486cc1f6dcd842d
    2019-07-10T01:28:43.872+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=21547fa1ebd63af111a824afee03c5420c5a7a5386ad22ae4b3935d679226c8a
    2019-07-10T01:28:44.221+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=95a901c2-b206-e7e3-16d4-3fd261a753bb task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:44.578+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=9d330a907ad2ba85ddf7862d68fe3563dc2e92153b513a2751af5a2aceefc7e6
    2019-07-10T01:28:44.933+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=0cef5646-5694-66c9-ef81-7f8a2485c478 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:45.410+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=0d327c9db89f6b7c48fbc396b47c4732e968f1fa6585c5021b85001eb73262a6
    2019-07-10T01:28:45.657+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=28f265d730560df730a7a32cbc20460a74cee7756a599ba24ac0933d8d5cd965
    2019-07-10T01:28:45.761+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=53530227-5dd1-93a4-28ae-70575bc33162 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:45.823+0300 [INFO ] client.driver_mgr.docker: started container: driver=docker container_id=624f9ab9f0f054078ee7021af978716321a7155efc771126136704deaddf77b4
    2019-07-10T01:28:45.836+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=9ffb644d-cade-7deb-c2c2-ef5b1961ccdb task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:46.162+0300 [INFO ] client: node registration complete
    2019-07-10T01:28:46.208+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=f570bc6e-3b0e-3876-f9db-3d6ad7a6a3e6 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:46.356+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=793e0894-26a9-d9b4-56f7-755e097d226a task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:46.703+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=79d2ff9a-3825-be3a-385b-137a2fa40186 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:46.846+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=50120b73-c417-c782-4b85-651128935a58 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:47.015+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=8775342c-6060-f733-b356-981820e71501 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:47.138+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=9f3cd741-b145-6596-d440-2a2cee8ef8c9 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:47.291+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=36688829-cf14-3c3b-0fce-f06853bc4b45 task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
    2019-07-10T01:28:47.429+0300 [INFO ] client.alloc_runner.task_runner: not restarting task: alloc_id=31f62ca4-5757-cfc0-a808-39ac99535eea task=IqoqoTask reason="Restart unnecessary as task terminated successfully"
@ilyaiqoqo ilyaiqoqo reopened this Jul 9, 2019
@ilyaiqoqo ilyaiqoqo changed the title restarting agent reruns successfully completed allocations Restarting agent reruns successfully completed allocations Jul 9, 2019
@preetapan
Copy link
Contributor

@ilyaiqoqo - This is addressed by #5890 which will land in our next upcoming point release Nomad 0.8.7.

@ilyaiqoqo
Copy link
Contributor Author

I can confirm it is fixed in master

@the-maldridge the-maldridge mentioned this issue Jul 10, 2019
Closed
@notnoop notnoop mentioned this issue Sep 15, 2020
Merged
@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 21, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
2 participants
@preetapan @ilyaiqoqo