You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
Output from nomad version Nomad v0.11.1 (825a2da27e778655a735804a084c0a131c6b6fec)
(has custom logging changes)
Issue
We noticed our EC2 instances (which only have a few gigs of disk) were running out of space, and narrowed the issue down to /var/lib/docker in spite of having the following GC settings in nomad
In reviewing the logs I noticed some times image de-referencing would happen, but the reference count the logs showed was the same as the count before, for example
In this case, the reference count was 3 after both de-references, within a second apart, and there was no reason anything else would have increased the ref count.
Delete may do nothing, if callerID is not found in references
And thus I made the following changes to the build to get more data
// Decrement the reference count
prevCount := len(references)
if _, ok := references[callerID]; !ok {
d.logger.Error("task ID is not contributing to ref count", "image_id", imageID, "task_id", callerID)
}
delete(references, callerID)
count := len(references)
d.logger.Info("image id reference count decremented", "image_id", imageID, "new_references", count, "old_references", prevCount, "references", references)
And made similar changes around reference count increase.
What I found was sometimes the caller ID seems to be in a different format than when it was passed to incrementImageReferenceImpl and so it does nothing
So we have 4 tasks running referencing the same image (0e88...).
3 of them end in -browser and 1 of them does not.
Now we stop all the tasks and expect reference count to hit 0
2020-05-12T22:48:42.683Z [ERROR] client.driver_mgr.docker: task ID is not contributing to ref count: driver=docker image_id=sha256:0e88eac525d26357ce48dd52ce78a40c88b7cb0353d2f8ba3210dfe624598def task_id=8379c08c-ec41-f5e7-1649-2a8fe071a12c/browser/b221394c
2020-05-12T22:48:42.683Z [INFO] client.driver_mgr.docker: image id reference count decremented: driver=docker image_id=sha256:0e88eac525d26357ce48dd52ce78a40c88b7cb0353d2f8ba3210dfe624598def new_references=4 old_references=4 references="map[8379c08c-ec41-f5e7-1649-2a8fe071a12c/browser/b221394c-browser:{} cebff5e5-5243-78e6-59c4-cfdbdc2cbd00/browser/dd84eb9d-browser:{} e3925196-1485-faa4-0693-7818c938d224/browser/353e434c-browser:{} ff93c709-3208-08ba-18c2-9c9b28c26d93/browser/20a6aef8:{}]"
Already we have a problem. This call to RemoveImage did nothing because the callerID passed in was 8379c08c-ec41-f5e7-1649-2a8fe071a12c/browser/b221394c however the callerID it should have passed in is 8379c08c-ec41-f5e7-1649-2a8fe071a12c/browser/b221394c-browser
Same tasks but different format - one has -browser after it and the other does not.
It does this 3 times, only 1 of them (the one that doesn't end in -browser in the map) has the correct task ID, resulting in a final reference count of 3 when it should be 0
The text was updated successfully, but these errors were encountered:
benbuzbee
changed the title
Docker drive leaks images
Docker driver leaks images
May 12, 2020
I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.
Nomad version
Output from
nomad version
Nomad v0.11.1 (825a2da27e778655a735804a084c0a131c6b6fec)
(has custom logging changes)
Issue
We noticed our EC2 instances (which only have a few gigs of disk) were running out of space, and narrowed the issue down to
/var/lib/docker
in spite of having the following GC settings in nomadIn reviewing the logs I noticed some times image de-referencing would happen, but the reference count the logs showed was the same as the count before, for example
In this case, the reference count was 3 after both de-references, within a second apart, and there was no reason anything else would have increased the ref count.
So I checked the code and realized the following:
Delete may do nothing, if
callerID
is not found inreferences
And thus I made the following changes to the build to get more data
And made similar changes around reference count increase.
What I found was sometimes the caller ID seems to be in a different format than when it was passed to incrementImageReferenceImpl and so it does nothing
Consider the following logs with my changes:
So we have 4 tasks running referencing the same image (0e88...).
3 of them end in
-browser
and 1 of them does not.Now we stop all the tasks and expect reference count to hit 0
Already we have a problem. This call to
RemoveImage
did nothing because the callerID passed in was8379c08c-ec41-f5e7-1649-2a8fe071a12c/browser/b221394c
however the callerID it should have passed in is8379c08c-ec41-f5e7-1649-2a8fe071a12c/browser/b221394c-browser
Same tasks but different format - one has -browser after it and the other does not.
It does this 3 times, only 1 of them (the one that doesn't end in -browser in the map) has the correct task ID, resulting in a final reference count of
3
when it should be0
The text was updated successfully, but these errors were encountered: