oadm prune is pruning layers referenced by other images

[aliscott]

Version

openshift v1.5.0
kubernetes v1.5.2+43a9be4
etcd 3.1.0

The image pruner is pruning layers from the registry that are referenced by other images. This is causing my deployment configs to fail with image back offs. When pulling the latest image I get an unknown blob error.

Here is the output after I had run oc adm prune images --keep-tag-revisions=5 --keep-younger-than=48h --confirm

Pruning images...
Deleting references from image streams to images ...
STREAM                                  IMAGE                                                                     TAGS
alistair/alistair-test                  sha256:7bf5ed0dc436d61376bfa024459eb025a4d2fc18bc6645844db964b294d17ba7   production
alistair/alistair-test                  sha256:356117288e5f22c63fda3678b1cc9ad87d57c43ddaa6f2a51ac4874aafb54621   latest
alistair/alistair-test                  sha256:569cdf0c53ade1dadedaaaba4137ef2723be261f586cc5ddb43e1527a0f6b876   production
alistair/alistair-test                  sha256:857fdbb8f59669d80cf3ed6e18f0f272024bcd5f1ed08b433691dd6531103f7d   latest

Deleting registry repository layer links ...
REPO                                    LAYER LINK
alistair/alistair-test                  sha256:62657e4e53ca403eb904624adbad6c9a5f2b20c151e8021a5587f9354741f621
alistair/alistair-test                  sha256:93eb8d61b8533304fc9e9f66b21eede40f5c389c9993b5da78256cd7e6f65a64
alistair/alistair-test                  sha256:94e84ca41deb47dd87935360bfbb1fff274d9be1de3a3df37e13ad83a5ed1eff
alistair/alistair-test                  sha256:8f9bc277260f6a8fce6896b14ae4d36a2a3177a60d1d080a16d0b8df3026839a
alistair/alistair-test                  sha256:09ca2dad26ed1383e82d8f0f1c95a9b247ae3eb878722bfe75ebde189adba2e9
alistair/alistair-test                  sha256:612637482224cdf1585b8679af4bbcbf3243c15cec7b0fd767dce0a16f624e49
alistair/alistair-test                  sha256:92fe130f38d42f36ba66d40538a90ae1e3037bff598cc36bbe531dd9373ca55e
alistair/alistair-test                  sha256:c8bb1cd45591beb7d97ccabeed553d8e9eb346716b800fc938dc58519592f916
alistair/alistair-test                  sha256:8c0826de39db8cbd8bc17ae94d9b79080a0f234385add05d2c756c07321fd76a
alistair/alistair-test                  sha256:344c434c62147aba3be6792f3fc2eed1d3aeb4d83a6366ab78fd16f9ff3cbc3f
alistair/alistair-test                  sha256:333b9f0ff5be009a101cc0038859b467002e8b98865220da9cd018985665a0ca
alistair/alistair-test                  sha256:1d3a88955d80d957214a0d22cb986960f24e02aaa906c408a29522fa15161652
alistair/alistair-test                  sha256:0b1977e4750613f611e121c6da030b6d2b473776c0ec08d32a7a2285cf794d73
alistair/alistair-test                  sha256:64cdeb1698f941c6195e45fc7c165314d276b0c6a1fceea8f8daa2c69cdd495b
alistair/alistair-test                  sha256:efaa9adc05c0c2e12532a3626badf9204eaf3c4b1a090c8c2ebc9960ad0a639c
alistair/alistair-test                  sha256:007ef6a94b5b9d25952ca80d366cfb88a6acb81c69e0df52a90bb566618a576c
alistair/alistair-test                  sha256:c13064d0de794ed8e7b38f451a83e32995995af585a82f34ac0e4f766a09652e
alistair/alistair-test                  sha256:5a0e3bc8e68222c4e3d1ca38706bd86689adc0ebab66b8060f165b0c2132afa6
alistair/alistair-test                  sha256:8e2544b8950a2a2054c0423f0a01d2670ede83c6c6debf78fe528550670d0a44
alistair/alistair-test                  sha256:c94bb4159c7bc299227ae4db13af4993a706d8b1061febad30f23c2e46725640
alistair/alistair-test                  sha256:85e1bbbdf51555712266069c2078a9190a805ebbd1fd802f88fb43c3959b30ca
alistair/alistair-test                  sha256:d5da99b0d01f6f352a8eaf248e528c2dd7dafab9586e238b9724ce48da69a291
alistair/alistair-test                  sha256:c93048b0a0d49e1a045bbcd8171cacd3341b7db54ab081917071776aabfa5849

Deleting registry layer blobs ...
BLOB
sha256:62657e4e53ca403eb904624adbad6c9a5f2b20c151e8021a5587f9354741f621
sha256:4932c1b1476fa1dcbf275b5acce4e2f5fa7085ee7574a3251b637a2e22d61b92
sha256:93eb8d61b8533304fc9e9f66b21eede40f5c389c9993b5da78256cd7e6f65a64
sha256:94e84ca41deb47dd87935360bfbb1fff274d9be1de3a3df37e13ad83a5ed1eff
sha256:8f9bc277260f6a8fce6896b14ae4d36a2a3177a60d1d080a16d0b8df3026839a
sha256:0676bc0591c0be710ac36695ed56327e26796a327e7835c993f7882db36412d2
sha256:54f902283746887f13d34c54ffcffeb76002b7ee99abc7820f0cc7607cbde7bf
sha256:09ca2dad26ed1383e82d8f0f1c95a9b247ae3eb878722bfe75ebde189adba2e9
sha256:b4136e55af068a3abb0926556badaa26262b48733150bd8f69d87c1b11073b1a
sha256:612637482224cdf1585b8679af4bbcbf3243c15cec7b0fd767dce0a16f624e49
sha256:92fe130f38d42f36ba66d40538a90ae1e3037bff598cc36bbe531dd9373ca55e
sha256:c8bb1cd45591beb7d97ccabeed553d8e9eb346716b800fc938dc58519592f916
sha256:8c0826de39db8cbd8bc17ae94d9b79080a0f234385add05d2c756c07321fd76a
sha256:344c434c62147aba3be6792f3fc2eed1d3aeb4d83a6366ab78fd16f9ff3cbc3f
sha256:333b9f0ff5be009a101cc0038859b467002e8b98865220da9cd018985665a0ca
sha256:e019220a8628a3f26822001589f4a41bc9f64186b4d7da9f7ba6c1df2d879f03
sha256:1d3a88955d80d957214a0d22cb986960f24e02aaa906c408a29522fa15161652
sha256:0b1977e4750613f611e121c6da030b6d2b473776c0ec08d32a7a2285cf794d73
sha256:64cdeb1698f941c6195e45fc7c165314d276b0c6a1fceea8f8daa2c69cdd495b
sha256:efaa9adc05c0c2e12532a3626badf9204eaf3c4b1a090c8c2ebc9960ad0a639c
sha256:007ef6a94b5b9d25952ca80d366cfb88a6acb81c69e0df52a90bb566618a576c
sha256:c13064d0de794ed8e7b38f451a83e32995995af585a82f34ac0e4f766a09652e
sha256:3b87a97d3d774f87724797fe3c5a53fde82419266a51fbcbbd2d96f44a182b0e
sha256:5a0e3bc8e68222c4e3d1ca38706bd86689adc0ebab66b8060f165b0c2132afa6
sha256:f94d94d7a6c18cc9dc28d64fe143cbc4dd00317d06830cc53ec1e65fa41c5344
sha256:f07dac2d0daa87ae38ba175ace4f2973d13084a4cb53c4781e8587a319dcac76
sha256:8e2544b8950a2a2054c0423f0a01d2670ede83c6c6debf78fe528550670d0a44
sha256:c94bb4159c7bc299227ae4db13af4993a706d8b1061febad30f23c2e46725640
sha256:eb3b37bd2a82c688d3cf5ecf3d52ff9f2db979411b0f17687e3418e2b9eebab1
sha256:dfc4a4817cb010d0f882f75d90ac1bc337eb63623db841f5adc0a181f91b09aa
sha256:85e1bbbdf51555712266069c2078a9190a805ebbd1fd802f88fb43c3959b30ca
sha256:d868e7d8ac77591c1d48224abbe71c9df2da8dfa526b92fe52c8ff3abbc69282
sha256:42828f577add8e966898f46f7af3b0259112e7416ffd81020df8799bccd601fa
sha256:d5da99b0d01f6f352a8eaf248e528c2dd7dafab9586e238b9724ce48da69a291
sha256:98c5eb7fa677244178db3c32d3acb39d7033ec2f1acde76c529e9c8dadc838b2
sha256:c93048b0a0d49e1a045bbcd8171cacd3341b7db54ab081917071776aabfa5849

Deleting registry repository manifest data ...
REPO                                    IMAGE
alistair/alistair-test                  sha256:7bf5ed0dc436d61376bfa024459eb025a4d2fc18bc6645844db964b294d17ba7
alistair/alistair-test                  sha256:356117288e5f22c63fda3678b1cc9ad87d57c43ddaa6f2a51ac4874aafb54621
alistair/alistair-test                  sha256:569cdf0c53ade1dadedaaaba4137ef2723be261f586cc5ddb43e1527a0f6b876
alistair/alistair-test                  sha256:857fdbb8f59669d80cf3ed6e18f0f272024bcd5f1ed08b433691dd6531103f7d

Output of docker pull for the latest sha:

$ docker pull registry.local/alistair/alistair-test@sha256:112edbdddc113654404d039fcb2984834834f385720d1f43f89266e6d1532bf5: Pulling from alistair/alistair-test
cd0a524342ef: Already exists
e39c3ffe4133: Already exists
f444da5fa2cd: Already exists
69ee8980874a: Already exists
d5c8431f000e: Already exists
e4b75d719b75: Already exists
efaa9adc05c0: Downloading
62657e4e53ca: Downloading
0b1977e47506: Downloading
85e1bbbdf515: Downloading
344c434c6214: Downloading
43772286ec1a: Download complete
unknown blob

Notice how it can't pull layer efaa9adc05c0 which had been pruned above.

Additional Information

Here is the output of oc get is alistair-test -o yaml

apiVersion: v1
kind: ImageStream
metadata:
  annotations:
    openshift.io/image.dockerRepositoryCheck: 2017-05-23T10:10:41Z
  creationTimestamp: 2017-04-29T08:39:01Z
  generation: 7
  name: alistair-test
  namespace: alistair
  resourceVersion: "25780943"
  selfLink: /oapi/v1/namespaces/alistair/imagestreams/alistair-test
  uid: 4b366732-2cb7-11e7-b158-00163e000872
spec:
  tags:
  - annotations: null
    from:
      kind: DockerImage
      name: alistair/alistair-test/prodcution
    generation: 5
    importPolicy: {}
    name: latest
    referencePolicy:
      type: Source
  - annotations: null
    from:
      kind: ImageStreamImage
      name: alistair-test@sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
      namespace: alistair
    generation: 7
    importPolicy: {}
    name: old-production
    referencePolicy:
      type: Source
  - annotations: null
    from:
      kind: ImageStreamImage
      name: alistair-test@sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
      namespace: alistair
    generation: 6
    importPolicy: {}
    name: production
    referencePolicy:
      type: Source
status:
  dockerImageRepository: 172.30.150.55:5000/alistair/alistair-test
  tags:
  - items:
    - created: 2017-06-23T14:55:20Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:112edbdddc113654404d039fcb2984834834f385720d1f43f89266e6d1532bf5
      generation: 7
      image: sha256:112edbdddc113654404d039fcb2984834834f385720d1f43f89266e6d1532bf5
    - created: 2017-06-21T11:16:44Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:a6a02ac93c3b0b25336c94f7338270a2c118dd7dfe96cefa2491b0608aefda43
      generation: 7
      image: sha256:a6a02ac93c3b0b25336c94f7338270a2c118dd7dfe96cefa2491b0608aefda43
    - created: 2017-06-21T10:32:03Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:09380108b53fdf583778110213095a900283cfb5f2e1ef9fbfc1b9e1ac34bbf5
      generation: 7
      image: sha256:09380108b53fdf583778110213095a900283cfb5f2e1ef9fbfc1b9e1ac34bbf5
    - created: 2017-06-21T10:23:52Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:732c05a4953ec2a75675091dca578536b962276b0836577a34897026d38ba936
      generation: 7
      image: sha256:732c05a4953ec2a75675091dca578536b962276b0836577a34897026d38ba936
    - created: 2017-06-21T06:58:01Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:faa10c10d39325fbed6681675993ce0f9bbb2096f5ee3c13884c5f0cb2f04d75
      generation: 7
      image: sha256:faa10c10d39325fbed6681675993ce0f9bbb2096f5ee3c13884c5f0cb2f04d75
    - created: 2017-06-21T06:43:58Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:ea0c694010f9b64643018ed579e4dee77befcd04692af4724e7706aee7d14761
      generation: 7
      image: sha256:ea0c694010f9b64643018ed579e4dee77befcd04692af4724e7706aee7d14761
    - created: 2017-06-21T06:41:05Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:94aba6dd5919cfb3024d3186f79f21a80c27394b869574c4f9b932821c1fca92
      generation: 7
      image: sha256:94aba6dd5919cfb3024d3186f79f21a80c27394b869574c4f9b932821c1fca92
    - created: 2017-06-07T06:56:16Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:b04f08273ef07cfe09dca4dd3b04e3c7c0e963d5ece23e148e44f2c54f295a25
      generation: 7
      image: sha256:b04f08273ef07cfe09dca4dd3b04e3c7c0e963d5ece23e148e44f2c54f295a25
    - created: 2017-06-07T06:48:50Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:349dea05543f87b4f37c53e21c4a1a9187cca350b179c0955fdd4ba138fe6513
      generation: 7
      image: sha256:349dea05543f87b4f37c53e21c4a1a9187cca350b179c0955fdd4ba138fe6513
    - created: 2017-06-06T15:07:03Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:99b060a03f7d7a1885572cd55f91f7fb557e9146ddebcbd48234c5eabe6b8ad7
      generation: 7
      image: sha256:99b060a03f7d7a1885572cd55f91f7fb557e9146ddebcbd48234c5eabe6b8ad7
    - created: 2017-05-23T06:03:55Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
      generation: 1
      image: sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
    tag: latest
  - items:
    - created: 2017-05-23T10:22:32Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
      generation: 7
      image: sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
    tag: old-production
  - items:
    - created: 2017-06-21T11:16:49Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:c320330c0120bd43e550c1b88465fa881bd458ec11803958edb810fd7a3b6d6f
      generation: 7
      image: sha256:c320330c0120bd43e550c1b88465fa881bd458ec11803958edb810fd7a3b6d6f
    - created: 2017-06-21T10:32:05Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:93ae8a17b2eeac08fcc802e1abc50e4a59f1bb16ab0e544a329f126ed1de4226
      generation: 7
      image: sha256:93ae8a17b2eeac08fcc802e1abc50e4a59f1bb16ab0e544a329f126ed1de4226
    - created: 2017-06-21T10:25:40Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:1133e2bb02c0a5b9b075984b753ec7879c91418e2202634a316aa1cc1b7bb93a
      generation: 7
      image: sha256:1133e2bb02c0a5b9b075984b753ec7879c91418e2202634a316aa1cc1b7bb93a
    - created: 2017-06-21T06:57:59Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:a9a2b82f4da9fbc77bef1ec465d5acd776d7b6b55ecde557fa80939789efd316
      generation: 7
      image: sha256:a9a2b82f4da9fbc77bef1ec465d5acd776d7b6b55ecde557fa80939789efd316
    - created: 2017-06-21T06:41:34Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:4e9cee6b838bc0ac86db9794bc8e23d53a9e2151239a9f468f614ab39a145129
      generation: 7
      image: sha256:4e9cee6b838bc0ac86db9794bc8e23d53a9e2151239a9f468f614ab39a145129
    - created: 2017-06-18T09:06:44Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:a7cb7d5a02f495a5e8930c4f228e3fdaf1ae801ef1279c0b7bbe330a38a3f5bc
      generation: 7
      image: sha256:a7cb7d5a02f495a5e8930c4f228e3fdaf1ae801ef1279c0b7bbe330a38a3f5bc
    - created: 2017-05-23T10:19:46Z
      dockerImageReference: 172.30.150.55:5000/alistair/alistair-test@sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
      generation: 6
      image: sha256:524f8b7d0aa0af85806d919349028c146dcead33dc564af3bfd403cd7e0c31d9
    tag: production

[php-coder]

@legionus @miminar @dmage

[dmage]

Was the image sha256:112edbdddc113654404d039fcb2984834834f385720d1f43f89266e6d1532bf5 pushed concurrently with the pruning?

[aliscott]

@dmage I don't have the exact timestamp from that prune anymore, but it seems likely that it did. Our pruning job runs at :55 past the hour and that image was created at 2017-06-23T14:55:20.

[aliscott]

Although saying that image sha256:a6a02ac93c3b0b25336c94f7338270a2c118dd7dfe96cefa2491b0608aefda43 also referenced the same layers and has the same issue and was created before the prune job ran.

[dmage]

@soltysh have you seen such a problem before?

[soltysh]

@dmage hmm... yeah, we've had a similar bug a while ago, see #13671. And the version @aliscott reports should have that fix.
I'd recommend looking into pruning logs, if it's reproducible, and get the exact commands that lead to this. It might be that something is still not right.

[dmage]

@aliscott can you provide the exact commands to reproduce this?

[aliscott]

@dmage I don't have a way of reproducing it 100% of the time, but I have seen it a couple of times now. We have a job that runs this every hour:

oc adm prune images --keep-tag-revisions=5 --keep-younger-than=48h --confirm
oc adm prune images --prune-over-size-limit --confirm

[alikhajeh1]

Any update on this?

[dmage]

It seems we have problems with keep-younger-than. No estimations, though.

[alikhajeh1]

Thanks for the update @dmage, so is --keep-tag-revisions safe to use?

[aliscott]

@dmage --keep-tag-revisions will work for me just now. Are you able to confirm the issue is only there when using --keep-younger-than?

[mrobson]

@aliscott @alikhajeh1 Yes -

Two images referenced Layer X:

Image A -> Layer X
Image B -> Layer X

oc adm prune Runs:

Image A is valid prunable
Image B is under minimum pruning age still (--keep-younger-than)

Result:
Image A is pruned with Layer X because Image B is not added to the graph (due to it's age). In this case, Layer X has no valid image predecessor via Image B.

[miminar]

@mrobson thanks for the confirmation! We've just found the problem: https://github.com/openshift/origin/pull/16580/files#r141612326