Docker driver removing container image on task failure

[stevenscg]

Nomad version

Nomad v0.12.0 (8f7fbc8)

Operating system and Environment details

CentOS 7.8.2003

Issue

As of v0.12.0, container images are incorrectly removed from the host when a task fails and docker.cleanup.image is set to false.

If the task starts, runs and then exits properly, this issue does not appear to occur.

The same jobs did not appear to exhibit this behavior on any of the prior releases. Version 0.11.3 was the most recent used prior to 0.12.0.

As shown in the config example, we have also recently tried quoting the client options keys and values per the documentation without any apparent change in the behavior described by this issue. We typically just quote the values in client options.

The use case for retaining the container images is a development environment where the same image is used for several days at a time. The setup scripts for this environment take care of building the container images, so users of this environment see jobs that fail to start randomly now that would have continued to work with past versions of Nomad.

Reproduction steps

Config file:

client {
  enabled = true
  options {
    driver.whitelist = "docker"
    "docker.cleanup.image" = "false"
  }
}

plugin "docker" {
  config {
    volumes {
      enabled = true
    }
  }
}

We are not certain what kind of task failure causes this behavior at this time. A simple task that exits with code 1 or similar may be sufficient.

[notnoop]

Thank you for reporting the issue. We'll investigate and follow up!

FWIW, between 0.9.0 and 0.11.2, Nomad wasn't GCing any images! We fixed image GC in #7947 in 0.11.2 - so I'm little surprised you didn't encounter this in 0.11.3 - so we'll need to dig in further.

[stevenscg]

@notnoop Noted. Thanks. I believe we were running 0.11.3 most recently because we tend to keep very current for this development environment. However, it could have been 0.11.2.

[stevenscg]

I believe that this issue is still present on v0.12.3.

[stevenscg]

@notnoop Is there any known or potential workaround for this kind of issue that I could try in the interim?

[stevenscg]

I spent some time with versions 0.11.4 and 0.11.2 on my test instance where this problem was occurring. These versions have the same behavior as 0.12.3, so I'm not entirely sure what's happening.

As a workaround, I tried using a long cleanup image delay which would be fine for my use case. It did not seem to make any difference in the errant behavior.

client {
  enabled = true
  ....
  options {
    "docker.auth.config" = "/etc/docker/config.json"
    "driver.whitelist" = "docker"
    "docker.cleanup.image" = "true"
    "docker.cleanup.image.delay" = "30m"
  }
}

docker version
Client: Docker Engine - Community
 Version:           19.03.12
 API version:       1.40
 Go version:        go1.13.10
 Git commit:        48a66213fe
 Built:             Mon Jun 22 15:46:54 2020
 OS/Arch:           linux/amd64
 Experimental:      false

consul version
Consul v1.8.3
Revision a9322b9c7
Protocol 2 spoken by default, understands 2 to 3 (agent will automatically use protocol >2 when speaking to compatible agents)

[stevenscg]

Still seeing this on v0.12.4.

[notnoop]

Hi @stevenscg, I'm very very sorry for taking a very long to investigate this. The issue seems to be mixing the old deprecated syntax with the new one. If plugin "docker" { config { ... } } is present in the config, the docker driver ignores the old client options fields. As you pointed out in the work around, once you use the options exclusively, the options are interpreted as expected.

I'd suggest adopting the new plugin config syntax with config like the following:

client {
  enabled = true
  options {
    driver.whitelist       = "docker"
  }
}

plugin "docker" {
  config {
    volumes {
      enabled = true
    }

    gc {
      image = false
    }
  }
}

Let me know if that addresses the issue!

[stevenscg]

@notnoop Thanks for the info! I had moved to the plugin syntax for "volumes" but not yet "gc", so I think this will fix my particular issue. I'll drop back in a few days and close it if it all checks out.

[stevenscg]

This all looks good, thanks! Closing.

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.