Fix caching to respect .dockerignore #854

WanzenBug · 2019-11-11T17:03:44Z

Description
Previously kaniko would compute the cache key for any copy command by computing
the combined hash of all files in a directory, even if they were listed
as ignored.

With this change, the cache key creation was updated to be aware of ignored
files.

Related issues:

Fixes Changes to files in .dockerignore invalidate COPY cache key #594

Submitter Checklist

These are the criteria that every PR should meet, please check them off as you
review them:

Includes unit tests
Adds integration tests if needed.

See the contribution guide for more details.

Reviewer Notes

The code flow looks good.
Unit tests and or integration tests added.

Release Notes

- caching knows about files excluded via `.dockerignore`

googlebot · 2019-11-11T17:03:58Z

Thanks for your pull request. It looks like this may be your first contribution to a Google open source project (if not, look below for help). Before we can look at your pull request, you'll need to sign a Contributor License Agreement (CLA).

📝 Please visit https://cla.developers.google.com/ to sign.

Once you've signed (or fixed any issues), please reply here with @googlebot I signed it! and we'll verify it.

What to do if you already signed the CLA

Individual signers

It's possible we don't have your GitHub username or you're using a different email address on your commit. Check your existing CLA data and verify that your email is set on your git commits.

Corporate signers

Your company has a Point of Contact who decides which employees are authorized to participate. Ask your POC to be added to the group of authorized contributors. If you don't know who your Point of Contact is, direct the Google project maintainer to go/cla#troubleshoot (Public version).
The email used to register you as an authorized contributor must be the email used for the Git commit. Check your existing CLA data and verify that your email is set on your git commits.
The email used to register you as an authorized contributor must also be attached to your GitHub account.

ℹ️ Googlers: Go here for more info.

WanzenBug · 2019-11-11T17:04:28Z

I'll try to add unit tests and integration tests when I have the time

WanzenBug · 2019-11-11T17:05:38Z

@googlebot I signed it!

googlebot · 2019-11-11T17:05:43Z

CLAs look good, thanks!

ℹ️ Googlers: Go here for more info.

tejal29 · 2019-11-14T23:23:28Z

@WanzenBug Thanks for your contribution. Would be up for adding tests?

WanzenBug · 2019-11-17T20:11:19Z

I've added unit tests to check that the AddPath method is working as expected

WanzenBug · 2019-11-20T20:13:54Z

@tejal29 I've added some unit tests

cvgw · 2019-12-20T20:31:39Z

Rebased

cvgw · 2019-12-21T02:10:30Z

docker ignore integration test failed

--- FAIL: TestLayers/test_layer_Dockerfile_test_dockerignore (6.09s)
        integration_test.go:308: Error building image: Failed to build image gcr.io/kaniko-test/docker-dockerfile_test_dockerignore with kaniko command "[docker run -v /home/kbuilder/.config/gcloud:/root/.config/gcloud -v /tmpfs/tmp/586476378:/kaniko/benchmarks -v /usr/local/go/src/github.com/GoogleContainerTools/kaniko/integration:/workspace -e BENCHMARK_FILE=false executor-image -f /workspace/dockerfiles/Dockerfile_test_dockerignore -d gcr.io/kaniko-test/kaniko-dockerfile_test_dockerignore  -c /workspace]": exit status 1 INFO[0000] Resolved base name scratch to scratch
            INFO[0000] Using dockerignore file: /workspace/.dockerignore
            INFO[0000] Resolved base name scratch to scratch
            INFO[0000] Built cross stage deps: map[]
            INFO[0000] No base image, nothing to extract
            INFO[0000] Unpacking rootfs as cmd COPY ignore/* /foo requires it.
            INFO[0000] Taking snapshot of full filesystem...
            INFO[0000] Resolving srcs [ignore/*]...
            error building image: error building stage: file does not exist

Previously kaniko would compute the cache key for any copy command by computing the combined hash of all files in a directory, even if they were listed as ignored. With this change, the cache key creation was updated to be aware of ignored files. Related issues: * GoogleContainerTools#594

Add unit tests that check the behaviour of CompositeCache on adding filesytem resources. It checks that * 2 identical directory trees produces the same hash * an extra file produces a different hash * an extra directry produces a different hash * an extra file that is excluded does not alter the hash * an extra directory that is excluded does not alter the hash

tejal29 · 2020-01-23T23:39:24Z

Rebasing this again.
Integration tests now run in Travis and @WanzenBug you can see the failing test now :)

Let us know if you need help finishing up this PR

tejal29 · 2020-01-24T22:03:19Z

pkg/executor/composite_cache.go

 	if err := filepath.Walk(p, func(path string, fi os.FileInfo, err error) error {
 		if err != nil {
 			return err
 		}
 		exclude := util.ExcludeFile(path, context)
-		if fi.IsDir() && exclude {


This is still needed right so we can skip walking whitelisted directories?

Actually this was onr if the bugs:
If the directory was ignored, but one the contained files was included via later rules in the ignore file, this would skip over this file completely.

The change now would still not include the directory in the cache key, but would include the contained file.

tejal29 added the needs-unit-tests label Nov 14, 2019

WanzenBug force-pushed the ignored-files-invalidate-cache branch from 44ad2c6 to 526ec29 Compare November 17, 2019 21:24

michaelhixson mentioned this pull request Nov 21, 2019

.dockerignore not considered when computing cache keys #871

Closed

tejal29 removed the needs-unit-tests label Dec 9, 2019

cvgw force-pushed the ignored-files-invalidate-cache branch from 268b948 to 9e0f466 Compare December 21, 2019 00:29

googlebot added the cla: yes CLA signed by all commit authors label Dec 21, 2019

cvgw added the kokoro:run label Dec 21, 2019

kokoro-team removed the kokoro:run label Dec 21, 2019

tejal29 added the integration-test-failed label Jan 17, 2020

WanzenBug and others added 3 commits January 23, 2020 15:37

update build and composite cache tests

782e491

tejal29 force-pushed the ignored-files-invalidate-cache branch from a6d8c88 to 782e491 Compare January 23, 2020 23:38

fix skipping ignored directories if they have whitelisted content

bab1c17

tejal29 reviewed Jan 24, 2020

View reviewed changes

tejal29 added candidate-release kokoro:run and removed integration-test-failed labels Jan 24, 2020

kokoro-team removed the kokoro:run label Jan 28, 2020

tejal29 approved these changes Jan 29, 2020

View reviewed changes

tejal29 merged commit 8bdcb4f into GoogleContainerTools:master Jan 29, 2020

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Fix caching to respect .dockerignore #854

Fix caching to respect .dockerignore #854

WanzenBug commented Nov 11, 2019

googlebot commented Nov 11, 2019

WanzenBug commented Nov 11, 2019

WanzenBug commented Nov 11, 2019

googlebot commented Nov 11, 2019

tejal29 commented Nov 14, 2019

WanzenBug commented Nov 17, 2019

WanzenBug commented Nov 20, 2019

cvgw commented Dec 20, 2019 •

edited

Loading

cvgw commented Dec 21, 2019

tejal29 commented Jan 23, 2020

tejal29 Jan 24, 2020

WanzenBug Jan 25, 2020

Fix caching to respect .dockerignore #854

Fix caching to respect .dockerignore #854

Conversation

WanzenBug commented Nov 11, 2019

googlebot commented Nov 11, 2019

What to do if you already signed the CLA

Individual signers

Corporate signers

WanzenBug commented Nov 11, 2019

WanzenBug commented Nov 11, 2019

googlebot commented Nov 11, 2019

tejal29 commented Nov 14, 2019

WanzenBug commented Nov 17, 2019

WanzenBug commented Nov 20, 2019

cvgw commented Dec 20, 2019 • edited Loading

cvgw commented Dec 21, 2019

tejal29 commented Jan 23, 2020

tejal29 Jan 24, 2020

Choose a reason for hiding this comment

WanzenBug Jan 25, 2020

Choose a reason for hiding this comment

cvgw commented Dec 20, 2019 •

edited

Loading