nomad monitor always panics

[m1kc]

Nomad version

Nomad v0.10.2 (0d2d6e3)

Operating system and Environment details

$ cat /etc/os-release 
NAME="Arch Linux"
PRETTY_NAME="Arch Linux"
ID=arch
BUILD_ID=rolling
ANSI_COLOR="0;36"
HOME_URL="https://www.archlinux.org/"
DOCUMENTATION_URL="https://wiki.archlinux.org/"
SUPPORT_URL="https://bbs.archlinux.org/"
BUG_REPORT_URL="https://bugs.archlinux.org/"
LOGO=archlinux

Issue

nomad monitor always fails with the following output, no matter what log-level and node-id I use:

$ nomad monitor -log-level=INFO -node-id=3d651214-cc4c-fe06-fe5c-9834ef11148c
panic: runtime error: index out of range

goroutine 1 [running]:
github.com/hashicorp/nomad/command.(*MonitorCommand).Run(0xc00002cf20, 0xc0000b8060, 0x2, 0x2, 0x0)
	/opt/gopath/src/github.com/hashicorp/nomad/command/agent_monitor.go:114 +0x1192
github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli.(*CLI).Run(0xc0004c68c0, 0xc0004c68c0, 0xc0002e0500, 0x38)
	/opt/gopath/src/github.com/hashicorp/nomad/vendor/github.com/mitchellh/cli/cli.go:255 +0x1f1
main.RunCustom(0xc0000b8050, 0x3, 0x3, 0xc00008e058)
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:138 +0x4a1
main.Run(...)
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:83
main.main()
	/opt/gopath/src/github.com/hashicorp/nomad/main.go:79 +0x63

Some nodes in the cluster are still on 0.9, but I don't try to get logs from them.

Reproduction steps

Call nomad monitor with any arguments.

Job file (if appropriate)

Nomad Client logs (if appropriate)

Nomad Server logs (if appropriate)

No new log lines appear after that command.

[m1kc]

any arguments

I decided to re-check if this part is accurate. If called without arguments at all, nomad monitor succeeds and starts streaming logs from current node. It only fails if called with -node-id, even if the node specified is the same one.

[endocrimes]

It appears that this happens when there are no nodes found for the provided node-id, e.g if there is a typo or you're requesting from the wrong region.

#6828 should fix the problem.

[m1kc]

Well, but the ID above does exist (I just copied and pasted it), and I have only one region here.

...Wait. Does that mean I can only request logs from client nodes?

[endocrimes]

I've filled #6845 as a follow up for Server node support. Sorry about that, it's confusing in our docs.

[endocrimes]

@m1kc - Oops, I missed that we have a -server-id flag that accepts the server name or leader.

[m1kc]

Thanks for pointing that out!

And... well, it looks like I still can't get it to work with server node. So, it still works fine without arguments, streaming logs from current node, it works with -node-id=<client node id> just as well, but with server nodes - no luck. For some reason, it returns code 400 every time:

$ nomad monitor -server-id=3d651214-cc4c-fe06-fe5c-9834ef11148c -log-level=DEBUG

==> Error starting monitor: Unexpected response code: 400 (unknown nomad server 3d651214-cc4c-fe06-fe5c-9834ef11148c)
==> For additional help try 'nomad monitor -help'

It's the same id you see on the screenshot above. I double-checked that it's actually a server id (and not a client id), it's actually valid, the server did not change its id for some reason, I'm pointing at the right machine and it actually has Nomad v0.10.2 (0d2d6e3) installed.

I can't actually try that with another 0.10.x server (for now, 'cause the other ones are still on 0.9.x), so can't say if the problem is only with this one. But, just for the sake of experiment, I tried requesting logs from these 0.9 servers anyway, and it returns exactly the same error message with code 400. Here are these responses too, in case I'm missing something again:

$ nomad monitor -server-id=35e5ed05-ed6d-9376-ee92-4a2e2356dd28 -log-level=DEBUG
==> Error starting monitor: Unexpected response code: 400 (unknown nomad server 35e5ed05-ed6d-9376-ee92-4a2e2356dd28)
==> For additional help try 'nomad monitor -help'

$ nomad monitor -server-id=96b5c3e4-423b-b326-02f9-d4aa9751db47 -log-level=DEBUG
==> Error starting monitor: Unexpected response code: 400 (unknown nomad server 96b5c3e4-423b-b326-02f9-d4aa9751db47)
==> For additional help try 'nomad monitor -help'

Now, as you could probably tell, I still feel kinda stupid after missing that -server-id flag in the first place, but... well, what am I doing wrong this time?

[drewbailey]

Hi @m1kc sorry for the confusion. Currently the -server-id flag only supports the servers name or a special leader value to target the current leader (docs). I've filed #6847 to allow passing in the servers actual ID was well.

Based off of your screen shot the following should work

nomad monitor -server-id=ocean.global -log-level=DEBUG

[m1kc]

-server-id flag only supports the servers name

That was... unexpected. But it worked, thanks!

[drewbailey]

I agree it's confusing and will be fixed in #6847. Sorry for the inconvenience, glad to hear its working now.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.