You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.
If filing a bug please include the following:
Nomad version
Nomad v0.3.2
Issue
I just encountered an issue where a Nomad agent in our cluster became unresponsive (I believe related to docker). Unable to SSH into the instance I force rebooted in the Google Cloud console. Nomad was unable to startup on reboot (panic) with the following stack trace:
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: Starting nomad...
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: Started nomad.
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: Loaded configuration from /etc/nomad.d/config.json
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: ==> Starting Nomad agent...
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: [signal 0xb code=0x1 addr=0x78 pc=0x5643f8]
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: goroutine 1 [running]:
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: panic(0x1124d20, 0xc82000e0c0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/go/src/runtime/panic.go:464 +0x3e6
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client/driver.GetTaskEnv(0xc8202e0810, 0xc8200bcfc0, 0x0, 0xc8201e90e0, 0x0, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/driver/driver.go:145 +0xf8
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*TaskRunner).setTaskEnv(0xc820243580, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:203 +0x5a
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*TaskRunner).RestoreState(0xc820243580, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:140 +0x122
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*AllocRunner).RestoreState(0xc8202b9c70, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:129 +0x593
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*Client).restoreState(0xc8201e0240, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:429 +0x53b
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.NewClient(0xc8201e8000, 0xc8201e8000, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:171 +0x8b9
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.(*Agent).setupClient(0xc8201d8780, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:290 +0x173
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.NewAgent(0xc820167b80, 0x7f59d3c9f920, 0xc8201af920, 0xc8201d86e0, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:55 +0x247
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.(*Command).setupAgent(0xc8201aa320, 0xc820167b80, 0x7f59d3c9f920, 0xc8201af920, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:286 +0xc3
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc8201aa320, 0xc82000a1a0, 0x2, 0x2, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:389 +0x597
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc8200bc540, 0xc8200bc540, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:153 +0x5ee
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: main.RunCustom(0xc82000a190, 0x3, 0x3, 0xc820192fc0, 0xc820054058)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/main.go:49 +0x4d5
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: main.Run(0xc82000a190, 0x3, 0x3, 0x7f59d3c96028)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/main.go:15 +0x4c
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: main.main()
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/main.go:11 +0x60
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: nomad.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: Unit nomad.service entered failed state.
To fix the issue I ran rm -rf /var/nomad/* followed with sudo systemctl restart nomad.
The text was updated successfully, but these errors were encountered:
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
If you have a question, prepend your issue with
[question]
or preferably use the nomad mailing list.If filing a bug please include the following:
Nomad version
Nomad v0.3.2
Issue
I just encountered an issue where a Nomad agent in our cluster became unresponsive (I believe related to docker). Unable to SSH into the instance I force rebooted in the Google Cloud console. Nomad was unable to startup on reboot (panic) with the following stack trace:
To fix the issue I ran
rm -rf /var/nomad/*
followed withsudo systemctl restart nomad
.The text was updated successfully, but these errors were encountered: