Nomad agent panics with nil pointer deference

[ghost]

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Nomad v0.3.2

Issue

I just encountered an issue where a Nomad agent in our cluster became unresponsive (I believe related to docker). Unable to SSH into the instance I force rebooted in the Google Cloud console. Nomad was unable to startup on reboot (panic) with the following stack trace:

Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: Starting nomad...
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: Started nomad.
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: Loaded configuration from /etc/nomad.d/config.json
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: ==> Starting Nomad agent...
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: panic: runtime error: invalid memory address or nil pointer dereference
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: [signal 0xb code=0x1 addr=0x78 pc=0x5643f8]
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: goroutine 1 [running]:
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: panic(0x1124d20, 0xc82000e0c0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/go/src/runtime/panic.go:464 +0x3e6
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client/driver.GetTaskEnv(0xc8202e0810, 0xc8200bcfc0, 0x0, 0xc8201e90e0, 0x0, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/driver/driver.go:145 +0xf8
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*TaskRunner).setTaskEnv(0xc820243580, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:203 +0x5a
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*TaskRunner).RestoreState(0xc820243580, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/task_runner.go:140 +0x122
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*AllocRunner).RestoreState(0xc8202b9c70, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/alloc_runner.go:129 +0x593
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.(*Client).restoreState(0xc8201e0240, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:429 +0x53b
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/client.NewClient(0xc8201e8000, 0xc8201e8000, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/client/client.go:171 +0x8b9
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.(*Agent).setupClient(0xc8201d8780, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:290 +0x173
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.NewAgent(0xc820167b80, 0x7f59d3c9f920, 0xc8201af920, 0xc8201d86e0, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/agent.go:55 +0x247
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.(*Command).setupAgent(0xc8201aa320, 0xc820167b80, 0x7f59d3c9f920, 0xc8201af920, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:286 +0xc3
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/command/agent.(*Command).Run(0xc8201aa320, 0xc82000a1a0, 0x2, 0x2, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/command/agent/command.go:389 +0x597
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc8200bc540, 0xc8200bc540, 0x0, 0x0)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:153 +0x5ee
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: main.RunCustom(0xc82000a190, 0x3, 0x3, 0xc820192fc0, 0xc820054058)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/main.go:49 +0x4d5
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: main.Run(0xc82000a190, 0x3, 0x3, 0x7f59d3c96028)
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/main.go:15 +0x4c
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: main.main()
Jun 14 16:20:43 deploy-api-ylm1 nomad[972]: /opt/gopath/src/github.com/hashicorp/nomad/main.go:11 +0x60
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: nomad.service: main process exited, code=exited, status=2/INVALIDARGUMENT
Jun 14 16:20:43 deploy-api-ylm1 systemd[1]: Unit nomad.service entered failed state.

To fix the issue I ran rm -rf /var/nomad/* followed with sudo systemctl restart nomad.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.