Nomad memory leak and pulling images with force_pull set to false

[fluffyf-x]

Nomad version

Output from nomad version
Nomad v0.12.7 (6147cb578794cb2d0c35d68fe1791728a09bb081)

Operating system and Environment details

Distributor ID:	Ubuntu
Description:	Ubuntu 18.04.6 LTS
Release:	18.04
Codename:	bionic

Issue

1. Nomad has a memory leak which we created a cron job to systemctl restart nomad every week, is there a fix for the memory leak

2. Nomad force_pull according to docs is false by default, however we see errors from pulling images on restart.
   prev issue: Nomad force_pull=false default not working #11220

Reproduction steps

1. allow nomad to run for weeks for it to be visible (watch memory increase, its fairly substantial)
2. sig 9 a container with nomad restart policy and cut off network access (to disallow pulling of containers)

Expected Result

1. nomad does not leak memory
2. nomad uses local container rather than pulling from container registry

Actual Result

1. nomad consistently uses more memory until machine is exhausted
2. nomad (using docker) is pulling external container registry

Job file (if appropriate)

Nomad Server logs (if appropriate)

Nomad Client logs (if appropriate)

[lgfa29]

Thanks for the report @fluffyf-x.

Would you be able to test with a 1.x version of Nomad to see if the problem still persists?

[fluffyf-x]

@lgfa29 : we were not able to test with the newer version, is there any chance you have the PR for the fix?

[tgross]

Hi @fluffyf-x Nomad 1.2.4 included a fix for a goroutine leak which you can see in #11741. Nomad 0.12.7 is long out of support, so we strongly recommend upgrading to get security fixes. If you still see a leak on Nomad 1.2.4 or above, please let us know.

[fluffyf-x]

Thank you @tgross @Igfa22, we upgraded to 1.0.0 and will upgrade to 1.2.4 today and let you know. I will close this issue in a week with a conclusion if we no longer see the memory leak.

[fluffyf-x]

Closing issue as resolved, thank you team!

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.