-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Nomad v1.4.3 crashes with 'panic: runtime error: invalid memory address or nil pointer dereference' #17310
Comments
Hi @brian-regrow! A crash is definitely a bug. I've had a quick look at the server logs you sent and the crash is when we're constructing the information about the node for the response back to the client (ref The relevant redacted portion of your server logs is as follows:
My hunch is that during the scale in we deregistered a node from the state store concurrently with a fingerprint update from that same node. We'll investigate further and circle back here. Thanks! |
Nomad version
v1.4.3
Autoscaler version - v0.3.7
Operating system and Environment details
Ubuntu 20.04
GCP - 3 server cluster, @180 client nodes, @1300 allocations.
Servers are n2-standard-16 - CPUx16, 64Gb RAM, 200Gb SSD - Usage on Cpu, Memory, Network, Disk is all low @20% max.
Config files, logs leading to crash and startup logs attached.
Issue
Nomad server crashes, cluster loses a node.
The issue occurred as Nomad was scaling in a node class from 144 -> 126 over a period of 7m
But this may be a red herring as we regularly have large scaling events, shortly afterwards the impaired cluster scaled the node class from 150 -> 5 nodes over a period of 10m without issue.
Reproduction steps
Cannot reproduce, this is the first time.
Expected Result
Nomad server does not crash.
Actual Result
Nomad server crashes, cluster loses a node.
Job file (if appropriate)
Many jobs running - the large node class is running a CPU intensive batch job.
The drain strategy is empty_ignore_system
Nomad Server logs (if appropriate)
Will send logs and config files by email.
The text was updated successfully, but these errors were encountered: