Nomad force_pull=false default not working

[fluffyf-x]

Nomad version

Output from nomad version

Nomad v0.12.7 (6147cb578794cb2d0c35d68fe1791728a09bb081)

Operating system and Environment details

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

running docker with nomad

Issue

We've seen in production that nomad pulls containers everytime a job restarts, however the documentation states that force_pull is set to false by default

2021-09-21T18:00:03.440Z [ERROR] client.driver_mgr.docker: failed pulling container: driver=docker image_ref=containers.pkg.****.com/****/***/turboscan:b54001f115b6d665bb0142eb0decc08c8fda1e23 "https://containers.pkg.github.com/v2/github/turboscan/turboscan/manifests/b54001f115b6d665bb0142eb0decc08c8fda1e23":

Reproduction steps

kill a container to force a nomad restart

Expected Result

nomad pulls local images before fetching from remote

Actual Result

nomad pulls an image everytime

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[gabrieltz]

hello,

I think your problem may be with docker image garbage collection which happens after a short time.
You may want to try the following in your task stanza which delays the deletion of images

gc {
    image       = true
    image_delay = "40m"
    container   = true
}

there are more gc options you can configure, see https://www.nomadproject.io/docs/drivers/docker

[DerekStrickland]

Hi @fluffyf-x ,

Thx for using Nomad!

Can you try out the suggestion @gabrieltz suggested and let us know if that resolves your issue?

Thanks!

@DerekStrickland and the Nomad Team

PS. Thanks @gabrieltz for pitching in!

[fluffyf-x]

hi @gabrieltz & @DerekStrickland : we have gc image collection set to false

    gc {
      image = false
    }

I'll add the image_delay thank you both!

[DerekStrickland]

Hi @fluffyf-x ,

I'm going to go ahead and close this issue, but feel free to re-open if you still feel there's a bug.

• Derek

[fluffyf-x]

Thank you for the prompt response

[notnoop]

Can you check with the latest nomad version? Nomad 1.0.0 had a bug fix related to image cleanup/GCing in #9101 . Prior to the fix, the image will be GCed immediately on failure, and it will be pulled on next restart. @gabrieltz suggestion works around this bug.

I'm commenting here, to avoid polluting issue #11240 with image force_pull details. It seems independent from the memory leak. Thanks!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.