-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Executor is killed after client cannot fetch stats #1719
Comments
Hey, I don't think this has anything to do with stats. It looks like the executor is exiting and you are seeing the error first with stats. Are you guys killing the executor? |
Hi, |
Our job is wrapped by a python script. Here we see nomad client sends a kill signal to executor and our script since it's child process of executor. auditd traces:
process 28878 sends SIGKILL(a1=9) to process 12383(a1=305f hex) process 28878 sends SIGKILL(a1=9) to process 12357(a1=3045 hex) |
I've patched
We expect if nomad executor is unreachable due to unreliable network, it should still be running after this change. In other words, executor did not panic but only was unreachable. Results confirmed that assumption. After some time when bug occured multiple times.
Bug occurs on 09:44:23
Processes after the bug
Executor with pid |
After 10 minutes both executors got killed by a panic. I noticed that change causes a memory leak in Nomad. Probably not related to bug.
|
I also see this in the logs.
|
@kaskavalci Was the nomad client memory leaking or was it the executors, and this only happened after your patch? |
@dadgar It's after the patch, I didn't encounter a memory leak rather than that. |
I changed unix sockets into good-old TCP connection to test if it is from the unix sockets but now it's just a different connection error. Here is the log output with more traces I added.
|
@kaskavalci How are you reproducing this so consistently? It would be great if you could share reproduction steps, so that we can debug more easily |
@kaskavalci Thanks so much for the dump. There is definitely a memory leak:
|
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
v0.4.0
Operating system and Environment details
Linux (Not tested on other platforms)
Issue
Nomad client sends
SIGKILL
to executor and its spawned process when driver cannot fetch stats for the task. We have a cluster in which we experienced some slow network communications. We assumeproblem may arise from executor_linux.go which is the only place Stats returns an error. (We have disabled that part and problem still occurs.pidStats
does not return an error effectively)Currently, not returning an error from Stats#L281 seems working. We'll test further.Reproduction steps
We don't have an exact reproduction steps since it only happens in
one clusterdocker, AWS and custom clusters, after some time.Nomad Client logs
The text was updated successfully, but these errors were encountered: