-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Tasks that are signaled to restart fail due to logmon errors #5574
Comments
hey @benvanstaveren, Thanks for reporting this - We haven't been able to reproduce this with the Please could you share your nomad job file, and if at all possible, verbose logs from a nomad-client running the allocation during the time of the restart? If you don't feel comfortable sharing these on GitHub, you can also email nomad-oss-debug@hashicorp.com, which is a mailing list readable by all HashiCorp employees, but not by the public. Thank you! |
Sure, no problem :) It's actually happened to a system job that runs Telegraf to collect (of all things) Nomad metrics so, here is the job file (with a few items redacted) Job file:
Here are the log entries relevant to the above job being restarted and failing, unfortunately I can't enable verbose logging at the moment because this is our production environment, so this is the default logs that Nomad tosses into syslog for the time being.
Also as a thing to note, this only seems to happen on jobs with tasks that have |
I can, probably, switch the log level to debug on one of our production boxes and trigger the error, but it has to wait until night time in Europe or I'll have the entire support desk in my office with the pitchforks and torches. |
@benvanstaveren That would be very useful - I'm also based in the EU so I can pick this up first thing tomorrow if so. |
I have emailed it to the address mentioned above as well as the job file, since it's a rather large file and has too much info in it for me to redact. However, an item of interest is that the exact same task in another job (same image, same configuration) that does not run with userns_mode = "host" and network_mode = "host" in the docker driver config section worked fine and got restarted properly. |
Alright, email has been sent - hope it gives you some clues. It does seem to be tied to having either all or a combination of userns_mode, network_mode and privileged settings in the docker driver config for a task. |
Okay sorry for the amount of noise but here's what I've managed to fiddle together: network_mode/host_mode/privileged apparently isn't the culprit - or it may be, but: the job that restarted successfully had labels set in the docker config:
I have added this same set of labels to the job that failed (the one I sent the details for by email), tried to reproduce the problem, and nothing happened, they restarted fine just like the other one. |
@benvanstaveren while we keep investigating this, wanted to let you know that we made our logmon process a bit more reliable with some fixes in 0.9.1-rc, available at https://releases.hashicorp.com/nomad/0.9.1-rc1/. Would you be able to try this out and let us know if you are still seeing problems with restarts? |
I can try but it won't be this week, I'll have to do this over a weekend so I don't interrupt production :) |
@benvanstaveren Thank you! |
It seems that I'm experiencing the same problem on windows with both 0.90 and 0.9.1-rc1 |
Sorry folks, couldn't pull it off over the weekend, will see if I can make an attempt during a weeknight with the rc1. Do I need to update the entire cluster to rc1 or just the machines running the task that causes the issues? |
@benvanstaveren just the clients should be fine 👍 - thanks and no problem |
@mpodkolzin Do you happen to have logs from the clients and also task events involved? - Thanks! |
I noticed the rc1 has been released as 0.9.1 so I'll put in an upgrade and see if I can trigger the error, probably sometime this week so will inform you all of the result here :) |
@benvanstaveren btw - 0.9.1 included few fixes in addition to the ones in rc1 - namely, #5616 . Looking at your logs from earlier, I believe this is the case you are hitting. Please try it and let us know |
This issue will be auto-closed because there hasn't been any activity for a few months. Feel free to open a new one if you still experience this problem 👍 |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
0.9.0
Operating system and Environment details
Linux, amd64
Issue
Any task that is signaled to restart (due to template changes, etc.) fails to properly restart due to logmon rpc errors.
Reproduction steps
Run service or system task with a template set to change_mode = "restart" - change the template (or a secret the template renders), watch it blow.
Nomad Client logs (if appropriate)
Events from a failed allocation:
51 minutes | 1 day | Restart Signaled | | | 0
51 minutes | 0 seconds | Terminated | | | 0
51 minutes | 0 seconds | Restarting | | | 0
51 minutes | 0 seconds | Task hook failed | logmon: rpc error: code = Unavailable desc = transport is closing | | 0
51 minutes | 0 seconds | Not Restarting | Error was unrecoverable
The text was updated successfully, but these errors were encountered: