-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker logs growing infinitely #5782
Comments
This seems to be a side effect of the recent changes in how Nomad collects logs from the Docker driver. For the time being I configured Docker to rotate its files in /etc/docker/daemon.json
I know I can pass the logging config to the driver in the job file but it would be nice to be able to do so also in the client config, under the newly introduced plugin/docker/config stanza |
How to add it to a nomad task stanza:
|
does #5846 fix this issue? |
Oops, yes! Thanks for the reminder @camerondavison ! Please test out 0.9.4-rc1 if you were hit by this: https://groups.google.com/d/msg/nomad-tool/Zi2I7P-PWo0/yAPFPmgLDQAJ |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.9.1 (4b2bdbd)
Issue
Infinitely growing Docker logs. Docker stores the logs under the
dockerd --data-root
. Nomad seems to be collecting them separately and also performs the nice log rotation there. However, the actual source of the logs is never cleaned. We already have a file which grew to 116GB worth of access logs in a short time, here adu -h
snippet:117G /var/lib/docker/containers/dafbe8ab0f5aec724e49d80aee5ef77824838870460a0182e76c0924b45ae4ba/dafbe8ab0f5aec724e49d80aee5ef77824838870460a0182e76c0924b45ae4ba-json.log
Reproduction steps
Any job producing logs
The text was updated successfully, but these errors were encountered: