Thousands of /tmp/pluginXXXX files

[carlpett]

Nomad version

Nomad v0.5.6

Operating system and Environment details

Centos 7.2

Issue

We just noticed we have thousands of zero-byte files named like /tmp/plugin000227203 on some of our nomad clients.

Reproduction steps

Unclear.

Nomad Client logs (if appropriate)

Looking at some files' time-stamps:

# ls -l plugin* --sort=time --time-style=full-iso | head
srwxr-xr-x. 1 nomad nomad 0 2017-07-14 18:58:03.584312115 +0200 plugin913505122
srwxr-xr-x. 1 nomad nomad 0 2017-07-14 18:58:03.571312377 +0200 plugin804998223

Here are nomad and docker logs from this time:

docker: INFO: 2017/07/14 16:58:03.279156 GET /containers/json
nomad: 2017/07/14 18:58:03.402616 [ERR] client: dropping update to terminal alloc '072318f8-0ec3-74c9-b330-8077bc382c18'
nomad: 2017/07/14 18:58:03.402720 [ERR] client: dropping update to terminal alloc '73ae3940-a609-36f0-029d-db85d42aa20b'
nomad: 2017/07/14 18:58:03.402772 [ERR] client: dropping update to terminal alloc 'ea992d34-8e21-3d24-9a64-76b350b1bff1'
nomad: 2017/07/14 18:58:03.402821 [ERR] client: dropping update to terminal alloc 'e97b257e-df4e-8362-b1a2-bf1016a3a60e'
nomad: 2017/07/14 18:58:03.402887 [ERR] client: dropping update to terminal alloc 'a8dc546e-bbe0-3702-acfa-85843870ae86'
nomad: 2017/07/14 18:58:03.402931 [ERR] client: dropping update to terminal alloc 'b506050d-5620-59d7-0f10-1f16736769d9'
nomad: 2017/07/14 18:58:03.402988 [ERR] client: dropping update to terminal alloc '12a7f663-a33c-8146-9fbb-dc93f0e572cc'
nomad: 2017/07/14 18:58:03.402871 [ERR] client: dropping update to terminal alloc 'e7ee1e35-8b9a-b453-fe1c-f8d7f6951f44'
docker: INFO: 2017/07/14 16:58:03.404969 POST /containers/293718c5d564247109bc88333d8453269b179ee13b31c84976fd842e3fb4634f/stop?t=5
docker: INFO: 2017/07/14 16:58:03.453638 GET /containers/293718c5d564247109bc88333d8453269b179ee13b31c84976fd842e3fb4634f/json
nomad: 2017/07/14 18:58:03.521728 [INFO] driver.docker: stopped container 293718c5d564247109bc88333d8453269b179ee13b31c84976fd842e3fb4634f
docker: INFO: 2017/07/14 16:58:03.530215 POST /containers/293718c5d564247109bc88333d8453269b179ee13b31c84976fd842e3fb4634f/stop?t=0
docker: INFO: 2017/07/14 16:58:03.531662 DELETE /containers/293718c5d564247109bc88333d8453269b179ee13b31c84976fd842e3fb4634f?force=1&v=1
nomad: 2017/07/14 18:58:03.537544 [INFO] client: marking allocation c2f6a672-c0e3-38f4-97b8-8747090ccaf2 for GC
nomad: 2017/07/14 18:58:03.537741 [INFO] client: marking allocation c2f6a672-c0e3-38f4-97b8-8747090ccaf2 for GC
docker: INFO: 2017/07/14 16:58:03.539625 GET /images/oliver006/redis_exporter:v0.9.1/json
docker: INFO: 2017/07/14 16:58:03.587464 POST /containers/create?name=redis_exporter-dnk-cache-690a173e-8413-4bea-0722-b6d393734243
nomad: 2017/07/14 18:58:03.598590 [ERR] client: dropping update to terminal alloc 'b506050d-5620-59d7-0f10-1f16736769d9'
nomad: 2017/07/14 18:58:03.598619 [ERR] client: dropping update to terminal alloc 'a8dc546e-bbe0-3702-acfa-85843870ae86'
nomad: 2017/07/14 18:58:03.598630 [ERR] client: dropping update to terminal alloc 'e7ee1e35-8b9a-b453-fe1c-f8d7f6951f44'
nomad: 2017/07/14 18:58:03.598657 [ERR] client: dropping update to terminal alloc '072318f8-0ec3-74c9-b330-8077bc382c18'
nomad: 2017/07/14 18:58:03.598638 [ERR] client: dropping update to terminal alloc '73ae3940-a609-36f0-029d-db85d42aa20b'
nomad: 2017/07/14 18:58:03.598645 [ERR] client: dropping update to terminal alloc 'e97b257e-df4e-8362-b1a2-bf1016a3a60e'
nomad: 2017/07/14 18:58:03.598621 [ERR] client: dropping update to terminal alloc '12a7f663-a33c-8146-9fbb-dc93f0e572cc'
nomad: 2017/07/14 18:58:03.598715 [ERR] client: dropping update to terminal alloc 'ea992d34-8e21-3d24-9a64-76b350b1bff1'
nomad: 2017/07/14 18:58:03.602584 [INFO] driver.docker: created container d9b13fa1d574e91fd64e078588a583395aacee4ba82fe0159405912f74e64f2b
docker: INFO: 2017/07/14 16:58:03.602984 POST /containers/d9b13fa1d574e91fd64e078588a583395aacee4ba82fe0159405912f74e64f2b/start
nomad: 2017/07/14 18:58:03.614322 [WARN] client: error fetching stats of task redis_exporter-dik: stats collection hasn't started yet
dockerd: time="2017-07-14T18:58:03.648890390+02:00" level=error msg="collecting stats for e74db744df52e45b74098e88d969720311092ef76bdbc917bb5647696f96593e: rpc error: code = 2 desc = exit status 1: \"container with id e74db744df52e45b74098e88d969720311092ef76bdbc917bb5647696f96593e is not running\\n\""
nomad: 2017/07/14 18:58:03.806186 [ERR] client: dropping update to terminal alloc '072318f8-0ec3-74c9-b330-8077bc382c18'
nomad: 2017/07/14 18:58:03.806220 [ERR] client: dropping update to terminal alloc 'a8dc546e-bbe0-3702-acfa-85843870ae86'
nomad: 2017/07/14 18:58:03.806200 [ERR] client: dropping update to terminal alloc 'e97b257e-df4e-8362-b1a2-bf1016a3a60e'
nomad: 2017/07/14 18:58:03.806206 [ERR] client: dropping update to terminal alloc '73ae3940-a609-36f0-029d-db85d42aa20b'
nomad: 2017/07/14 18:58:03.806210 [ERR] client: dropping update to terminal alloc '12a7f663-a33c-8146-9fbb-dc93f0e572cc'
nomad: 2017/07/14 18:58:03.806215 [ERR] client: dropping update to terminal alloc 'b506050d-5620-59d7-0f10-1f16736769d9'
nomad: 2017/07/14 18:58:03.806195 [ERR] client: dropping update to terminal alloc 'ea992d34-8e21-3d24-9a64-76b350b1bff1'
nomad: 2017/07/14 18:58:03.806299 [ERR] client: dropping update to terminal alloc 'e7ee1e35-8b9a-b453-fe1c-f8d7f6951f44'

I'm not sure what a terminal alloc is? Possibly it is unrelated to the files, and those correspond to container creation?

[dadgar]

@carlpett Have these clients always been running 0.5.6 or they have been upgraded from older versions? Also have they been restarted a lot?

[carlpett]

They've been upgraded: 0.4.0->0.4.1->0.5.0->0.5.4->0.5.5->0.5.6. First install was a bit over a year ago.
I'm not sure about a lot of restarts, but sure, probably more than a couple of times. Quite a few issues in the earlier version required a restart to mitigate, as I recall. But not anything excessive, like several times per day if that is what you mean?

[dadgar]

@carlpett I didn't notice the timestamps when I first looked at the issue. Are the vast majority of the plugins files recent? There was a bug that was fixed in an older version that meant we weren't cleaning up the files, that is why I asked if the node has been upgraded.

[carlpett]

@dadgar Yep, almost all of them are from the last week or so.

[dadgar]

@carlpett I wonder if you just had a non-clean shutdown of Nomad that didn't let it clean itself up?

[dadgar]

@carlpett Ignore my questions, I verified that this is happening! Will hopefully get a fix in 0.6.1

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.