-
Notifications
You must be signed in to change notification settings - Fork 1.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Docker driver: sha256-specified images keep getting re-downloaded #9619
Comments
👋 Can you give a more complete reproduction scenario? I tried:
I did not see that the updated job caused the image to be redownloaded when looking at the allocation events, nor in the agent logs. It would be helpful to know what steps you're following to see an undesired image download. |
Hmm we're getting these redownloads on other images as well (non-sha256-specified), perhaps the driver is misconfigured somehow. For me it reproduces by simply stopping+restarting the job in the UI. We're using proper nomad clients/servers. I'll double check the driver config |
Hi @exFalso ! I wonder if that's an issue of image GC interactions, potentially exacerbated by the image_delay bug fixed in #9101. Once a task stops and the image isn't referenced by any other task, Nomad will GC it immediately (!!), and it will be re-downloaded again next time. In 0.12.2, you can configure the docker gc behavior by setting the client config plugin "docker" {
config {
# ... rest of config
gc {
# you can disable GC completely, hopefully with using another GC mechanism
image = false
# alternatively, set a delay so images are only deleted after being unused for a long time
image_delay = "1h"
}
} Let us know if that helps! |
We've tested the suggestion in #9619 (comment) and it worked perfectly, thank you! |
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues. |
Nomad version
v0.12.2
Operating system and Environment details
Arch Linux, NixOS
Issue
Causes nomad to keep invalidating the image as if the tag was
latest
causing spurious downloads on redeploy, even though this is the most specific way to refer to an image.The text was updated successfully, but these errors were encountered: