-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker driver removing container image on task failure #8552
Comments
Thank you for reporting the issue. We'll investigate and follow up! FWIW, between 0.9.0 and 0.11.2, Nomad wasn't GCing any images! We fixed image GC in #7947 in 0.11.2 - so I'm little surprised you didn't encounter this in 0.11.3 - so we'll need to dig in further. |
@notnoop Noted. Thanks. I believe we were running 0.11.3 most recently because we tend to keep very current for this development environment. However, it could have been 0.11.2. |
I believe that this issue is still present on v0.12.3. |
@notnoop Is there any known or potential workaround for this kind of issue that I could try in the interim? |
I spent some time with versions 0.11.4 and 0.11.2 on my test instance where this problem was occurring. These versions have the same behavior as 0.12.3, so I'm not entirely sure what's happening. As a workaround, I tried using a long cleanup image delay which would be fine for my use case. It did not seem to make any difference in the errant behavior.
|
Still seeing this on v0.12.4. |
Hi @stevenscg, I'm very very sorry for taking a very long to investigate this. The issue seems to be mixing the old deprecated syntax with the new one. If I'd suggest adopting the new plugin config syntax with config like the following: client {
enabled = true
options {
driver.whitelist = "docker"
}
}
plugin "docker" {
config {
volumes {
enabled = true
}
gc {
image = false
}
}
} Let me know if that addresses the issue! |
@notnoop Thanks for the info! I had moved to the plugin syntax for "volumes" but not yet "gc", so I think this will fix my particular issue. I'll drop back in a few days and close it if it all checks out. |
This all looks good, thanks! Closing. |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
Nomad v0.12.0 (8f7fbc8)
Operating system and Environment details
CentOS 7.8.2003
Issue
As of v0.12.0, container images are incorrectly removed from the host when a task fails and docker.cleanup.image is set to false.
If the task starts, runs and then exits properly, this issue does not appear to occur.
The same jobs did not appear to exhibit this behavior on any of the prior releases. Version 0.11.3 was the most recent used prior to 0.12.0.
As shown in the config example, we have also recently tried quoting the client options keys and values per the documentation without any apparent change in the behavior described by this issue. We typically just quote the values in client options.
The use case for retaining the container images is a development environment where the same image is used for several days at a time. The setup scripts for this environment take care of building the container images, so users of this environment see jobs that fail to start randomly now that would have continued to work with past versions of Nomad.
Reproduction steps
Config file:
We are not certain what kind of task failure causes this behavior at this time. A simple task that exits with code 1 or similar may be sufficient.
The text was updated successfully, but these errors were encountered: