How to prevent pulling for COPY --link --from=image-with-one-layer / / ?

[skirsten]

Pretty simple. I want to concat the single layer from one image to the existing layers of the current image.

This ☝️ is working fine and its reusing the layer on the registry level (same digest).

The problem is that buildkit still pulls the layer, extracts it, does the COPY and "preparing layers for inline cache".
After which it found that it basically just has to upload a new manifest and can link the existing layers.

This is taking forever and I want to figure out if there is some other way to concat the manifests without pulling the layers.

Otherwise I would have to manually use crane rebase and implement my own cache system, which I prefer not to do 😅

[tonistiigi]

The layer is not pulled unless there is some process that actually needs to read its contents. If you are doing a build with --push to a registry then it is not pulled if the image is from the same repository, or from the same registry and that registry implement cross-repo mounts. If it is a different registry then there is no way for buildkit to transfer layer between registries without actually pulling it down first.

[tonistiigi]

Sorry, I didn't understand that this case was different from our regular example cases that use cache-from.

COPY --link in Dockerfile is a combination of 2 steps: copy to empty layer and then merge that layer.

The copy part still needs to run and access files to do that. For a special case COPY --link / / (no extra flags except --from allowed) copy could be skipped but Dockerfile frontend does not have such optimization atm. We could experiment with external frontend for this feature as no new buildkit changes are needed.

@sipsma IIRC we left this out because if the source image has more than one layer then it would mess up the history array. Possibly if we limit the case to --from=image we could do a ResolveImageConfig request and check how many layers the image has. It does start to get quite complicated though.

When using --cache-from then this case does not exist as copy part can be verified to be unchanged as compared to the cache and skipped over.

[skirsten]

Thank you for the detailed explanation.
I added a second --cache-from pointing to a similar image that has the same or similar COPY step in hopes that this would reduce the number of times I run into this problem and it worked.

But now I have a different problem: The build is not reproducible:

For some reason this part of the moby.buildkit.cache.v0 gets removed randomly if I rebuild the image a few times.

"chains": [
  {
    "layers": [
      2
    ],
    "createdAt": "0001-01-01T00:00:00Z"
  }
],

Everything else stays the same. Once it is removed its never added back but as far as I can tell its not predictable.

Here is the table for "is chains included":

          img 1   img 2   img 3   img 4   img 5   img 6   img 7   img 8   img 9   img 10
build 1   yes     yes     yes     yes     yes     yes     yes     yes     yes     yes
build 2   yes     yes     yes     no      no      no      yes     yes     yes     no
build 3   no      yes     yes     no      no      no      yes     no      yes     no
build 4   no      no      yes     no      no      no      no      no      yes     no
build 5   no      no      yes     no      no      no      no      no      yes     no

as can be seen its always included in the first build but then randomly vanishes in a later build...

Any ideas or is this a bug?

[tonistiigi]

If you build with --cache-to type=inline then that means that your image needs to contain moby.buildkit.cache.v0. You can see from the progressbar if there is an "exporting inline cache" line there.

[skirsten]

Yes, I am aware that it needs to contain the cache. The problem is that after a few builds this cache value (base64) actually changes.
I don't know how to explain that but its super easy to reproduce (on latest and master buildkit):

test/Dockerfile that just copies something

FROM busybox@sha256:98de1ad411c6d08e50f26f392f3bc6cd65f686469b7c22a85c7b5fb1b820c154

# do some stupid copy
COPY --link --from=alpine@sha256:9b2a28eb47540823042a2ba401386845089bb7b62a9637d55816132c4c3c36eb /bin/ls /bin/ls

reproduce.sh

img="ghcr.io/skirsten/tmp:test-loop-4" # Update or remove this image between tests

for j in {1..10}; do
  buildctl prune >/dev/null

  echo "building..."
  buildctl build --frontend dockerfile.v0 --local context=test --local dockerfile=test \
    --import-cache type=registry,ref=$img \
    --export-cache type=inline \
    --output type=image,name=$img,push=true \
    --metadata-file metadata.json 2>/dev/null

  digest=$(jq -r '."containerimage.digest"' metadata.json)
  config_digest=$(jq -r '."containerimage.config.digest"' metadata.json)

  echo "digest: $digest"
  echo "config_digest: $config_digest"

  crane config "$img@$digest" | jq -r '."moby.buildkit.cache.v0"' | base64 -d | jq . >new.json

  diff old.json new.json || true
  mv new.json old.json

  echo

  sleep 1
done

Output:

building...
digest: sha256:630b2de48a2e51079cec38c002f0c8bc3820b859557963c9fbc79e0d4697ecb1
config_digest: sha256:7231e8e691bad124e4d170998c4b566487a1c61deeffa4a74c5c441138ca640f

building...
digest: sha256:630b2de48a2e51079cec38c002f0c8bc3820b859557963c9fbc79e0d4697ecb1
config_digest: sha256:7231e8e691bad124e4d170998c4b566487a1c61deeffa4a74c5c441138ca640f

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5
33,40d32
<     "chains": [
<       {
<         "layers": [
<           1
<         ],
<         "createdAt": "0001-01-01T00:00:00Z"
<       }
<     ],

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

building...
digest: sha256:9c703d1e490f1e6dfbfde75e6be0a1ee50ea9de7072e10b422c331820b1c50fe
config_digest: sha256:d915739f055f10545f797706423c52f88b5cddd429cb40640b45a03c0f33b8e5

as can be seen, the third build removes the chains object from the base64'd cache. This causes the config_digest to change which then changes the digest. At which build number it breaks seems random to me... So this breaks the reproducibility.

This seems like a bug to me...

[tonistiigi]

Open an issue for this description. I don't instantly understand what case would trigger this.

[skirsten]

I created a issue for the second topic here: #3009

Thanks for all your help @tonistiigi . I will resolve this discussion now.
Takeaways for the initial topic:

• I would love if a optimization for the specific COPY --link --from=image-with-one-layer / / could be added. But I get that its a niche feature.
• The workaround for me was to import a second cache that contains the same --from image in the COPY so that it can use the cache. This reduced the number of cases where it needs to pull and prepare cache for the full image by 1/5.