You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm running the AWS EFS CSI driver on the Nomad clients via Docker image amazon/aws-efs-csi-driver:v1.3.8 deployed as a system job.
During some testing the Docker container started crashing which caused in turn both the Nomad Server, and all the Nomad Clients to panic:
Nomad Server (leader at the time - Issue fixed in #15095)
2022-12-05T15:39:08.698Z [ERROR] nomad.volumes_watcher: error releasing volume claims: namespace=ns volume_id=datastore-uploads
error=
| 9 errors occurred:
| \t* could not detach from node: node detach volume: EOF
| \t* could not detach from node: node detach volume: EOF
| \t* could not detach from node: node detach volume: EOF
| \t* could not detach from node: rpc error: node detach volume: CSI.NodeDetachVolume: plugin aws-efs for type csi-node not found
| \t* could not detach from node: rpc error: node detach volume: EOF
| \t* could not detach from node: No path to node
| \t* could not detach from node: No path to node
| \t* could not detach from node: No path to node
| \t* could not detach from node: No path to node
|
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x110 pc=0x1d647f3]
goroutine 132161661 [running]:
github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).volumeReapImpl(0x0?, 0x0)
github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:208 +0x33
github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).volumeReap(0xc00488f2c0, 0xc002088ea0?)
github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:193 +0x58
github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).watch(0xc00488f2c0)
github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:121 +0x5f
created by github.com/hashicorp/nomad/nomad/volumewatcher.(*volumeWatcher).Start
github.com/hashicorp/nomad/nomad/volumewatcher/volume_watcher.go:89 +0x125
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Nomad version
Nomad v1.4.1+ent
Operating system and Environment details
Issue
I'm running the AWS EFS CSI driver on the Nomad clients via Docker image amazon/aws-efs-csi-driver:v1.3.8 deployed as a system job.
During some testing the Docker container started crashing which caused in turn both the Nomad Server, and all the Nomad Clients to panic:
Nomad Server (leader at the time - Issue fixed in #15095)
Nomad Client (all of them)
Reproduction steps
The trigger to the issue seems to be the sudden unavailability of the aws-efs plugin on all clients
Expected Result
No panic.
The text was updated successfully, but these errors were encountered: