panic in volume watcher

[tgross]

Reported through our support org:

panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x110 pc=0x1d647f3]

goroutine 22850374 [running]:
github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).volumeReapImpl(0x0?, 0x0)
    github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:208 +0x33
github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).volumeReap(0xc002c84f00, 0xc0020ea1a0?)
    github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:193 +0x58
github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).watch(0xc002c84f00)
    github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:121 +0x5f
created by github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).Start
    github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:89 +0x125

This is bubbling up from volume_watcher.go#L208, which suggests the volume is nil here. A quick look at the code shows we're missing a nil-check at volume_watcher.go#L120-L121

[tgross]

I've got a patch worked up for this that includes a test that shows we can hit this if the volumewatcher restarts its goroutine just as a delete happens. For example, if a GC claim is written and as a result the volume is deleted before the volumewatcher can enter the loop in the run function, it'll panic. There's a race condition here around shutting down the volumewatcher goroutine I need to solve before I can get that fix up though.

[tgross]

#15101 has been merged and will ship in the next release of Nomad, with backports to the 1.3.x and 1.2.x series.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.