Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

nomad node status -verbose panics when a running alloc uses a host volume #8901

Closed
notnoop opened this issue Sep 16, 2020 · 1 comment · Fixed by #8902
Closed

nomad node status -verbose panics when a running alloc uses a host volume #8901

notnoop opened this issue Sep 16, 2020 · 1 comment · Fixed by #8902

Comments

@notnoop
Copy link
Contributor

notnoop commented Sep 16, 2020

Nomad version

Output from `nomad version`
root@nomad-client-1:~# nomad version
Nomad v0.12.4 (8efaee4ba5e9727ab323aaba2ac91c2d7b572d84)

Operating system and Environment details

In https://play.instruqt.com/hashicorp/tracks/nomad-host-volumes - in the last exercise "Write Data to the MySQL Database`

Issue

If a running allocation on a node uses a Host Volume, then nomad node status -verbose panics.

The reason is that the running allocation reports the Host Volume, which nomad node status -verbose mistakes as a CSI volume. Then when the nomad CLI attempts to print the nil CSI volume, it panics. The relevant code is in https://github.com/hashicorp/nomad/blob/v0.12.4/command/node_status.go#L545-L593 .

Some observations

root@nomad-client-1:~# nomad alloc status  --json c916bd47 | jq '.Job.TaskGroups[0].Volumes'
{
  "mysql_volume": {
    "MountOptions": null,
    "Name": "mysql_volume",
    "ReadOnly": false,
    "Source": "mysql",
    "Type": "host"
  }
}
root@nomad-client-1:~# nomad node status  -self
ID              = dd635239-afef-c695-5fff-d59329f4984b
Name            = client1
Class           = <none>
DC              = dc1
Drain           = false
Eligibility     = eligible
Status          = ready
CSI Controllers = <none>
CSI Drivers     = <none>
Uptime          = 1h14m0s
Host Volumes    = mysql
CSI Volumes     = mysql_volume
Driver Status   = docker,exec,java

Node Events
Time                  Subsystem  Message
2020-09-16T14:56:23Z  Cluster    Node registered

Allocated Resources
CPU           Memory           Disk
500/2300 MHz  1.0 GiB/3.6 GiB  300 MiB/15 GiB

Allocation Resource Utilization
CPU         Memory
7/2300 MHz  370 MiB/3.6 GiB
Host Resource Utilization
CPU         Memory           Disk
0/2300 MHz  686 MiB/3.6 GiB  4.7 GiB/19 GiB

Allocations
ID        Node ID   Task Group    Version  Desired  Status   Created     Modified
c916bd47  dd635239  mysql-server  0        run      running  17m47s ago  17m4s ago
root@nomad-client-1:~# nomad node status --verbose -self
ID              = dd635239-afef-c695-5fff-d59329f4984b
Name            = client1
Class           = <none>
DC              = dc1
Drain           = false
Eligibility     = eligible
Status          = ready
CSI Controllers = <none>
CSI Drivers     = <none>
Uptime          = 1h14m8s

Host Volumes
Name   ReadOnly  Source
mysql  false     /opt/mysql/data

CSI Volumes
panic: runtime error: invalid memory address or nil pointer dereference
[signal SIGSEGV: segmentation violation code=0x1 addr=0x8 pc=0x25cc97b]

goroutine 1 [running]:
github.com/hashicorp/nomad/command.(*NodeStatusCommand).outputNodeCSIVolumeInfo(0xc000118ea0, 0xc0001fe850, 0xc0002c2fc0, 0x
c0003103f0, 0x1, 0x1)
        github.com/hashicorp/nomad/command/node_status.go:583 +0x6ab
github.com/hashicorp/nomad/command.(*NodeStatusCommand).formatNode(0xc000118ea0, 0xc0001fe850, 0xc0002c2fc0, 0x0)
        github.com/hashicorp/nomad/command/node_status.go:436 +0x1a93
github.com/hashicorp/nomad/command.(*NodeStatusCommand).Run(0xc000118ea0, 0xc00004e0d0, 0x2, 0x2, 0xc000123090)
        github.com/hashicorp/nomad/command/node_status.go:279 +0x87f
github.com/mitchellh/cli.(*CLI).Run(0xc0001cedc0, 0xc0001cedc0, 0xc0001ef5e0, 0x38)
        github.com/mitchellh/cli@v1.1.0/cli.go:260 +0x1da
main.RunCustom(0xc00004e0b0, 0x4, 0x4, 0xc0000a2058)
        github.com/hashicorp/nomad/main.go:142 +0x482
main.Run(...)
        github.com/hashicorp/nomad/main.go:87
main.main()
        github.com/hashicorp/nomad/main.go:83 +0x64

Sorry the instance died as my session expired before my data got more data.

@github-actions
Copy link

github-actions bot commented Nov 2, 2022

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.

@github-actions github-actions bot locked as resolved and limited conversation to collaborators Nov 2, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

2 participants