Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Nomad doesn't cleanup down nodes after gc time #1245

Closed
tantra35 opened this issue Jun 9, 2016 · 9 comments · Fixed by #1256
Closed

Nomad doesn't cleanup down nodes after gc time #1245

tantra35 opened this issue Jun 9, 2016 · 9 comments · Fixed by #1256
Labels
theme/core type/bug

Comments

@tantra35
Copy link
Contributor

tantra35 commented Jun 9, 2016 •
edited

Nomad version

0.3.2

Operating system and Environment details

Linux consulnomad01 3.13.0-74-generic #118-Ubuntu SMP Thu Dec 17 22:52:10 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux

in amazon aws

Issue

We have some down node which doesn't removed after gc time.

root@consulnomad01:/home/ruslan# nomad node-status
ID        DC   Name             Class     Drain  Status
4ff014ff  aws  aws020           township  false  ready
dd68f51a  aws  ip-172-30-0-126  nginx     false  ready
dcdc4583  aws  ip-172-30-0-127  nginx     false  ready
bc9ffe4e  aws  ip-172-30-0-107  nginx     false  down
f2cb3de4  aws  aws018           township  false  ready
4c7d7bc6  aws  aws019           township  false  ready
c54048fd  aws  aws015           township  false  ready
752d0aa4  aws  aws017           township  false  ready
37b7efaf  aws  ip-172-30-1-249  fishdom   false  down
9b4818e3  aws  aws020           township  false  down
497928a2  aws  ip-172-30-0-39   fishdom   false  ready

Nodes

bc9ffe4e
9b4818e3

Present in list more than 1 day

As mention in https://groups.google.com/forum/#!topic/nomad-tool/oQFPQSNrpxU, we provide follow info, for node bc9ffe4e(full ID bc9ffe4e-2331-9a01-191f-6b54f0d0e083) which present in list alredy 5 days

a). One of the node object which should be down

curl http://localhost:4646/v1/node/bc9ffe4e-2331-9a01-191f-6b54f0d0e083
{  
   "ID":"bc9ffe4e-2331-9a01-191f-6b54f0d0e083",
   "Datacenter":"aws",
   "Name":"ip-172-30-0-107",
   "HTTPAddr":"127.0.0.1:4646",
   "Attributes":{  
      "arch":"amd64",
      "consul.datacenter":"aws",
      "consul.revision":"26a0ef8c41aa2252ab4cf0844fc6470c8e1d8256",
      "consul.server":"false",
      "consul.version":"0.6.4",
      "cpu.frequency":"2400.074000",
      "cpu.modelname":"Intel(R) Xeon(R) CPU E5-2676 v3 @ 2.40GHz",
      "cpu.numcores":"2",
      "cpu.totalcompute":"4800.148000",
      "driver.exec":"1",
      "driver.raw_exec":"1",
      "hostname":"ip-172-30-0-107",
      "kernel.name":"linux",
      "kernel.version":"3.13.0-74-generic",
      "memory.totalbytes":"4143996928",
      "nomad.revision":"'86f36d40591b29ceedc71fad9f37bea130e46316+CHANGES'",
      "nomad.version":"0.3.2",
      "os.name":"ubuntu",
      "os.version":"14.04",
      "platform.aws.instance-type":"t2.medium",
      "platform.aws.placement.availability-zone":"us-east-1a",
      "unique.cgroup.mountpoint":"/sys/fs/cgroup",
      "unique.consul.name":"ip-172-30-0-107",
      "unique.network.ip-address":"172.30.0.91",
      "unique.platform.aws.ami-id":"ami-eab94687",
      "unique.platform.aws.hostname":"ip-172-30-0-91.ec2.internal",
      "unique.platform.aws.instance-id":"i-e982e673",
      "unique.platform.aws.local-hostname":"ip-172-30-0-91.ec2.internal",
      "unique.platform.aws.local-ipv4":"172.30.0.91",
      "unique.platform.aws.public-hostname":"ec2-52-90-130-187.compute-1.amazonaws.com",
      "unique.platform.aws.public-ipv4":"52.90.130.187",
      "unique.storage.bytesfree":"31204802560",
      "unique.storage.bytestotal":"33675436032",
      "unique.storage.volume":"/dev/disk/by-uuid/0a76513a-37fc-43df-9833-34f8f9598ada"
   },
   "Resources":{  
      "CPU":4800,
      "MemoryMB":3952,
      "DiskMB":29759,
      "IOPS":0,
      "Networks":[  
         {  
            "Device":"eth0",
            "CIDR":"172.30.0.91/32",
            "IP":"172.30.0.91",
            "MBits":16,
            "ReservedPorts":null,
            "DynamicPorts":null
         },
         {  
            "Device":"eth0",
            "CIDR":"172.30.0.91/32",
            "IP":"172.30.0.91",
            "MBits":100,
            "ReservedPorts":null,
            "DynamicPorts":null
         }
      ]
   },
   "Reserved":{  
      "CPU":0,
      "MemoryMB":0,
      "DiskMB":0,
      "IOPS":0,
      "Networks":null
   },
   "Links":{  
      "aws.ec2":"us-east-1a.i-e982e673",
      "consul":"aws.ip-172-30-0-107"
   },
   "Meta":{  

   },
   "NodeClass":"nginx",
   "ComputedClass":"v1:10375885918857020582",
   "Drain":false,
   "Status":"down",
   "StatusDescription":"",
   "CreateIndex":12890,
   "ModifyIndex":13063
}

b). The allocation objects(atleast one) which is associated with the down node. [/v1/node//allocations]

curl http://localhost:4646/v1/node/bc9ffe4e-2331-9a01-191f-6b54f0d0e083/allocations
[  
   {  
      "ID":"88cb2bfa-ad68-863f-b6a5-6924a45f7406",
      "EvalID":"74e4ddc1-ae48-9192-56ce-cc7ff5d64d40",
      "Name":"nginx.nginx[0]",
      "NodeID":"bc9ffe4e-2331-9a01-191f-6b54f0d0e083",
      "JobID":"nginx",
      "Job":{  
         "Region":"global",
         "ID":"nginx",
         "ParentID":"",
         "Name":"nginx",
         "Type":"service",
         "Priority":50,
         "AllAtOnce":false,
         "Datacenters":[  
            "aws"
         ],
         "Constraints":[  
            {  
               "LTarget":"${attr.kernel.name}",
               "RTarget":"linux",
               "Operand":"distinct_hosts"
            },
            {  
               "LTarget":"${node.class}",
               "RTarget":"nginx",
               "Operand":"="
            }
         ],
         "TaskGroups":[  
            {  
               "Name":"nginx",
               "Count":2,
               "Constraints":null,
               "RestartPolicy":{  
                  "Attempts":2,
                  "Interval":60000000000,
                  "Delay":15000000000,
                  "Mode":"delay"
               },
               "Tasks":[  
                  {  
                     "Name":"nginx",
                     "Driver":"raw_exec",
                     "User":"",
                     "Config":{  
                        "command":"/usr/sbin/nginx"
                     },
                     "Env":null,
                     "Services":null,
                     "Constraints":null,
                     "Resources":{  
                        "CPU":500,
                        "MemoryMB":1000,
                        "DiskMB":300,
                        "IOPS":0,
                        "Networks":null
                     },
                     "Meta":null,
                     "KillTimeout":5000000000,
                     "LogConfig":{  
                        "MaxFiles":3,
                        "MaxFileSizeMB":10
                     },
                     "Artifacts":null
                  },
                  {  
                     "Name":"diamond",
                     "Driver":"raw_exec",
                     "User":"",
                     "Config":{  
                        "args":[  
                           "-f",
                           "-l"
                        ],
                        "command":"/usr/bin/diamond"
                     },
                     "Env":null,
                     "Services":null,
                     "Constraints":null,
                     "Resources":{  
                        "CPU":100,
                        "MemoryMB":500,
                        "DiskMB":300,
                        "IOPS":0,
                        "Networks":null
                     },
                     "Meta":null,
                     "KillTimeout":5000000000,
                     "LogConfig":{  
                        "MaxFiles":3,
                        "MaxFileSizeMB":10
                     },
                     "Artifacts":null
                  },
                  {  
                     "Name":"elb",
                     "Driver":"raw_exec",
                     "User":"",
                     "Config":{  
                        "args":[  
                           "-b",
                           "httpslb"
                        ],
                        "command":"/opt/elbmonitor/elbmonitor"
                     },
                     "Env":null,
                     "Services":null,
                     "Constraints":null,
                     "Resources":{  
                        "CPU":100,
                        "MemoryMB":300,
                        "DiskMB":300,
                        "IOPS":0,
                        "Networks":null
                     },
                     "Meta":null,
                     "KillTimeout":5000000000,
                     "LogConfig":{  
                        "MaxFiles":3,
                        "MaxFileSizeMB":10
                     },
                     "Artifacts":null
                  }
               ],
               "Meta":null
            }
         ],
         "Update":{  
            "Stagger":10000000000,
            "MaxParallel":1
         },
         "Periodic":null,
         "Meta":null,
         "Status":"running",
         "StatusDescription":"",
         "CreateIndex":12904,
         "ModifyIndex":13056,
         "JobModifyIndex":13056
      },
      "TaskGroup":"nginx",
      "Resources":{  
         "CPU":700,
         "MemoryMB":1800,
         "DiskMB":900,
         "IOPS":0,
         "Networks":null
      },
      "TaskResources":{  
         "diamond":{  
            "CPU":100,
            "MemoryMB":500,
            "DiskMB":300,
            "IOPS":0,
            "Networks":null
         },
         "elb":{  
            "CPU":100,
            "MemoryMB":300,
            "DiskMB":300,
            "IOPS":0,
            "Networks":null
         },
         "nginx":{  
            "CPU":500,
            "MemoryMB":1000,
            "DiskMB":300,
            "IOPS":0,
            "Networks":null
         }
      },
      "Services":{  

      },
      "Metrics":{  
         "NodesEvaluated":1,
         "NodesFiltered":0,
         "NodesAvailable":null,
         "ClassFiltered":null,
         "ConstraintFiltered":null,
         "NodesExhausted":0,
         "ClassExhausted":null,
         "DimensionExhausted":null,
         "Scores":{  
            "bc9ffe4e-2331-9a01-191f-6b54f0d0e083.binpack":9.348533726410107
         },
         "AllocationTime":14312,
         "CoalescedFailures":0
      },
      "DesiredStatus":"stop",
      "DesiredDescription":"alloc is being migrated",
      "ClientStatus":"running",
      "ClientDescription":"",
      "TaskStates":{  
         "diamond":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464961628721726831,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464961628741575949,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         },
         "elb":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464961628721773558,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464961628751576555,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Terminated",
                  "Time":1464964965698563689,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":1,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Restarting",
                  "Time":1464964965698763207,
                  "RestartReason":"Restart within policy",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":16133175332,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464964981851275061,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         },
         "nginx":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464961628721676457,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464961628750558206,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         }
      },
      "CreateIndex":12917,
      "ModifyIndex":13067,
      "AllocModifyIndex":13067,
      "CreateTime":1464961628703639306
   }
]

c). The job object. [/v1/job/]

curl http://localhost:4646/v1/job/nginx
{  
   "Region":"global",
   "ID":"nginx",
   "ParentID":"",
   "Name":"nginx",
   "Type":"service",
   "Priority":50,
   "AllAtOnce":false,
   "Datacenters":[  
      "aws"
   ],
   "Constraints":[  
      {  
         "LTarget":"${attr.kernel.name}",
         "RTarget":"linux",
         "Operand":"distinct_hosts"
      },
      {  
         "LTarget":"${node.class}",
         "RTarget":"nginx",
         "Operand":"="
      }
   ],
   "TaskGroups":[  
      {  
         "Name":"nginx",
         "Count":2,
         "Constraints":null,
         "RestartPolicy":{  
            "Attempts":2,
            "Interval":60000000000,
            "Delay":15000000000,
            "Mode":"delay"
         },
         "Tasks":[  
            {  
               "Name":"nginx",
               "Driver":"raw_exec",
               "User":"",
               "Config":{  
                  "command":"/usr/sbin/nginx"
               },
               "Env":null,
               "Services":null,
               "Constraints":null,
               "Resources":{  
                  "CPU":500,
                  "MemoryMB":1000,
                  "DiskMB":300,
                  "IOPS":0,
                  "Networks":null
               },
               "Meta":null,
               "KillTimeout":5000000000,
               "LogConfig":{  
                  "MaxFiles":3,
                  "MaxFileSizeMB":10
               },
               "Artifacts":null
            },
            {  
               "Name":"diamond",
               "Driver":"raw_exec",
               "User":"",
               "Config":{  
                  "args":[  
                     "-f",
                     "-l"
                  ],
                  "command":"/usr/bin/diamond"
               },
               "Env":null,
               "Services":null,
               "Constraints":null,
               "Resources":{  
                  "CPU":100,
                  "MemoryMB":500,
                  "DiskMB":300,
                  "IOPS":0,
                  "Networks":null
               },
               "Meta":null,
               "KillTimeout":5000000000,
               "LogConfig":{  
                  "MaxFiles":3,
                  "MaxFileSizeMB":10
               },
               "Artifacts":null
            },
            {  
               "Name":"elb",
               "Driver":"raw_exec",
               "User":"",
               "Config":{  
                  "args":[  
                     "-b",
                     "httpslb"
                  ],
                  "command":"/opt/elbmonitor/elbmonitor"
               },
               "Env":null,
               "Services":null,
               "Constraints":null,
               "Resources":{  
                  "CPU":100,
                  "MemoryMB":300,
                  "DiskMB":300,
                  "IOPS":0,
                  "Networks":null
               },
               "Meta":null,
               "KillTimeout":5000000000,
               "LogConfig":{  
                  "MaxFiles":3,
                  "MaxFileSizeMB":10
               },
               "Artifacts":null
            }
         ],
         "Meta":null
      }
   ],
   "Update":{  
      "Stagger":10000000000,
      "MaxParallel":1
   },
   "Periodic":null,
   "Meta":null,
   "Status":"running",
   "StatusDescription":"",
   "CreateIndex":12904,
   "ModifyIndex":13056,
   "JobModifyIndex":13056
}
@dadgar
Copy link
Contributor

dadgar commented Jun 10, 2016

Hey,

If you run curl -XPUT http://127.0.0.1:4646/v1/system/gc two times in a row does the node or the allocation on the node go away? Can you paste the server logs from that?

@dadgar
Copy link
Contributor

dadgar commented Jun 10, 2016 •
edited

Can you also paste server logs of the normal periodic gc. The lines should look like:

[DEBUG] sched.core: eval GC: scanning before index 3311 (1 hour)
[DEBUG] sched.core: eval GC: 2 evaluations, 3 allocs eligible

Same for node and job

@tantra35
Copy link
Contributor Author

tantra35 commented Jun 10, 2016 •
edited

my cmd output

root@consulnomad01:~# nomad node-status
ID        DC   Name             Class     Drain  Status
4ff014ff  aws  aws020           township  false  ready
dd68f51a  aws  ip-172-30-0-126  nginx     false  ready
dcdc4583  aws  ip-172-30-0-127  nginx     false  ready
bc9ffe4e  aws  ip-172-30-0-107  nginx     false  down
f2cb3de4  aws  aws018           township  false  ready
4c7d7bc6  aws  aws019           township  false  ready
c54048fd  aws  aws015           township  false  ready
752d0aa4  aws  aws017           township  false  ready
9b4818e3  aws  aws020           township  false  down
497928a2  aws  ip-172-30-0-39   fishdom   false  ready
root@consulnomad01:~# curl -XPUT http://127.0.0.1:4646/v1/system/gc
root@consulnomad01:~# curl -XPUT http://127.0.0.1:4646/v1/system/gc
root@consulnomad01:~# nomad node-status
ID        DC   Name             Class     Drain  Status
4ff014ff  aws  aws020           township  false  ready
dd68f51a  aws  ip-172-30-0-126  nginx     false  ready
dcdc4583  aws  ip-172-30-0-127  nginx     false  ready
bc9ffe4e  aws  ip-172-30-0-107  nginx     false  down
f2cb3de4  aws  aws018           township  false  ready
4c7d7bc6  aws  aws019           township  false  ready
c54048fd  aws  aws015           township  false  ready
752d0aa4  aws  aws017           township  false  ready
9b4818e3  aws  aws020           township  false  down
497928a2  aws  ip-172-30-0-39   fishdom   false  ready

Finally we restart nomad with debug log output, so get follow logs

Jun 10 20:55:52 consulnomad01 nomad[26589]: http: Request /v1/system/gc (1.406459ms)
Jun 10 20:55:54 consulnomad01 nomad[26589]: http: Request /v1/system/gc (1.554037ms)
Jun 10 20:56:02 consulnomad01 nomad[26589]: worker: dequeued evaluation 7ef784d4-1865-2fd3-0b92-3f98a4789134
Jun 10 20:56:02 consulnomad01 nomad[26589]: sched.core: eval GC: scanning before index 24539 (1h0m0s)
Jun 10 20:56:02 consulnomad01 nomad[26589]: worker: ack for evaluation 7ef784d4-1865-2fd3-0b92-3f98a4789134

so two times in a row doesn't help. То collect normal gc, we need some time

@dadgar
Copy link
Contributor

dadgar commented Jun 10, 2016

Can you also run curl http://127.0.0.1:4646/v1/evaluation/74e4ddc1-ae48-9192-56ce-cc7ff5d64d40/allocations

@dadgar dadgar mentioned this issue Jun 11, 2016
Merged
@tantra35
Copy link
Contributor Author 

curl http://127.0.0.1:4646/v1/evaluation/74e4ddc1-ae48-9192-56ce-cc7ff5d64d40/allocations

[  
   {  
      "ID":"88cb2bfa-ad68-863f-b6a5-6924a45f7406",
      "EvalID":"74e4ddc1-ae48-9192-56ce-cc7ff5d64d40",
      "Name":"nginx.nginx[0]",
      "NodeID":"bc9ffe4e-2331-9a01-191f-6b54f0d0e083",
      "JobID":"nginx",
      "TaskGroup":"nginx",
      "DesiredStatus":"stop",
      "DesiredDescription":"alloc is being migrated",
      "ClientStatus":"running",
      "ClientDescription":"",
      "TaskStates":{  
         "diamond":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464961628721726831,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464961628741575949,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         },
         "elb":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464961628721773558,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464961628751576555,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Terminated",
                  "Time":1464964965698563689,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":1,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Restarting",
                  "Time":1464964965698763207,
                  "RestartReason":"Restart within policy",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":16133175332,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464964981851275061,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         },
         "nginx":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464961628721676457,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464961628750558206,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         }
      },
      "CreateIndex":12917,
      "ModifyIndex":13067,
      "CreateTime":1464961628703639306
   },
   {  
      "ID":"c3b7e338-bc67-e890-1af0-23a0e0d1b322",
      "EvalID":"74e4ddc1-ae48-9192-56ce-cc7ff5d64d40",
      "Name":"nginx.nginx[1]",
      "NodeID":"dd68f51a-3767-adca-8725-71bb6b7ef51e",
      "JobID":"nginx",
      "TaskGroup":"nginx",
      "DesiredStatus":"run",
      "DesiredDescription":"",
      "ClientStatus":"running",
      "ClientDescription":"",
      "TaskStates":{  
         "diamond":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464965561632723011,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464965561670434841,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         },
         "elb":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464965561632764254,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464965561666798018,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         },
         "nginx":{  
            "State":"running",
            "Events":[  
               {  
                  "Type":"Received",
                  "Time":1464965561632664297,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               },
               {  
                  "Type":"Started",
                  "Time":1464965562050748683,
                  "RestartReason":"",
                  "DriverError":"",
                  "ExitCode":0,
                  "Signal":0,
                  "Message":"",
                  "KillError":"",
                  "StartDelay":0,
                  "DownloadError":"",
                  "ValidationError":""
               }
            ]
         }
      },
      "CreateIndex":13046,
      "ModifyIndex":13058,
      "CreateTime":1464965561610847115
   }
]

Also we publish our normal gc launch, as promise before:

Jun 11 06:46:02 consulnomad01 nomad[26589]: worker: dequeued evaluation f5d8c07a-6cf5-b2cc-b30d-dcce05895e25
Jun 11 06:46:02 consulnomad01 nomad[26589]: sched.core: eval GC: scanning before index 25294 (1h0m0s)
Jun 11 06:46:02 consulnomad01 nomad[26589]: worker: dequeued evaluation 16d0b7c9-c830-d13d-189e-c7e3510ae00b
Jun 11 06:46:02 consulnomad01 nomad[26589]: sched.core: node GC: scanning before index 23771 (24h0m0s)
Jun 11 06:46:02 consulnomad01 nomad[26589]: worker: ack for evaluation f5d8c07a-6cf5-b2cc-b30d-dcce05895e25
Jun 11 06:46:02 consulnomad01 nomad[26589]: worker: ack for evaluation 16d0b7c9-c830-d13d-189e-c7e3510ae00b

@dadgar
Copy link
Contributor

dadgar commented Jun 11, 2016

The PR I just submitted should fix this!

@tantra35
Copy link
Contributor Author

But how fix this with nomad 0.3.2? Any workaround? Or we must upgrade to last master ?

@dadgar
Copy link
Contributor

dadgar commented Jun 12, 2016

You can not with 0.3.2. I would wait for 0.4 which will be out soon. This will have no performance impact as the object is very light weight.

@github-actions
Copy link

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Dec 22, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
theme/core type/bug
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

Improve partial garbage collection of allocations hashicorp/nomad
2 participants
@dadgar @tantra35