Docker driver: sha256-specified images keep getting re-downloaded

[exFalso]

Nomad version

v0.12.2

Operating system and Environment details

Arch Linux, NixOS

Issue

      task "sometask" {                                                                                                                                              
	driver = "docker"                                                                                                                                                 
        config {                                                                                                                                                          
	  image = "https://somecr.io/repo/someimage@sha256:ab12c3ab12c3ab12c3ab12c3..." 
          ...                                                                 
        }
        ...
      }

Causes nomad to keep invalidating the image as if the tag was latest causing spurious downloads on redeploy, even though this is the most specific way to refer to an image.

[krishicks]

👋 Can you give a more complete reproduction scenario?

I tried:

$ nomad job init
Example job file written to example.nomad
$ sed -i 's/:3.2/@sha256:7b0a40301bc1567205e6461c5bf94c38e1e1ad0169709e49132cafc47f6b51f3/' example.nomad
$ nomad agent -dev

# in a new terminal
$ nomad job run example.nomad
$ sed '358ienv { foo = "bar" }' example.nomad | nomad job run -

I did not see that the updated job caused the image to be redownloaded when looking at the allocation events, nor in the agent logs.

It would be helpful to know what steps you're following to see an undesired image download.

[exFalso]

Hmm we're getting these redownloads on other images as well (non-sha256-specified), perhaps the driver is misconfigured somehow. For me it reproduces by simply stopping+restarting the job in the UI. We're using proper nomad clients/servers. I'll double check the driver config

[notnoop]

Hi @exFalso ! I wonder if that's an issue of image GC interactions, potentially exacerbated by the image_delay bug fixed in #9101. Once a task stops and the image isn't referenced by any other task, Nomad will GC it immediately (!!), and it will be re-downloaded again next time.

In 0.12.2, you can configure the docker gc behavior by setting the client config gc block in the client config:

plugin "docker" {
  config {
    # ... rest of config

    gc {
      # you can disable GC completely, hopefully with using another GC mechanism
      image       = false

      # alternatively, set a delay so images are only deleted after being unused for a long time
      image_delay = "1h"
    }
}

Let us know if that helps!

[exFalso]

We've tested the suggestion in #9619 (comment) and it worked perfectly, thank you!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.