Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Corrupted cache #16

Merged
merged 38 commits into from
Jul 17, 2024
Merged

Corrupted cache #16

merged 38 commits into from
Jul 17, 2024

Conversation

joeybrown-sf
Copy link
Collaborator

Summary

LOW-2: Denial-of-Service (DoS) provoked by removing build cache tarballs or altering the OCI image manifest

This PR addresses the security concern by skipping the cache restore if the tarball is not present.

A user will see the a similar warning as the following. The restore no longer hard fails.

Warning: Layer with SHA 'sha256:5a1185b2a8f6e19dfbd7db44d1cd31e67873798bc2a38eb3c3df5bba733d0eb4' not found, skipping restore

Release notes

Restorer no longer fails when missing cache artifacts are referenced in the cache metadata.


Related

Resolves buildpacks/lifecycle#1382


Context

Screenshot 2024-07-02 at 3 38 22 PM

natalieparellano and others added 12 commits April 22, 2024 15:06
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
…ot, (#1348)

consider it not a match

Signed-off-by: Natalie Arellano <narellano@vmware.com>
… labels (#1347)

* Read `/etc/os-release` file when distro information is not present in labels

Signed-off-by: Pavel Busko <pavel.busko@sap.com>

* fix unit tests

Signed-off-by: Pavel Busko <pavel.busko@sap.com>

* Update phase/generator_test.go

Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Pavel Busko <busko.pavel@gmail.com>

* Update phase/generator_test.go

Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Pavel Busko <busko.pavel@gmail.com>

---------

Signed-off-by: Pavel Busko <pavel.busko@sap.com>
Signed-off-by: Pavel Busko <busko.pavel@gmail.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Bumps the go-dependencies group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) | `1.21.1` | `1.22.0` |
| [github.com/containerd/containerd](https://github.com/containerd/containerd) | `1.7.14` | `1.7.16` |
| [github.com/docker/docker](https://github.com/docker/docker) | `26.0.1+incompatible` | `26.1.0+incompatible` |
| [github.com/moby/buildkit](https://github.com/moby/buildkit) | `0.13.1` | `0.13.2` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.18.0` | `0.19.0` |



Updates `github.com/GoogleContainerTools/kaniko` from 1.21.1 to 1.22.0
- [Release notes](https://github.com/GoogleContainerTools/kaniko/releases)
- [Changelog](https://github.com/GoogleContainerTools/kaniko/blob/main/CHANGELOG.md)
- [Commits](GoogleContainerTools/kaniko@v1.21.1...v1.22.0)

Updates `github.com/containerd/containerd` from 1.7.14 to 1.7.16
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](containerd/containerd@v1.7.14...v1.7.16)

Updates `github.com/docker/docker` from 26.0.1+incompatible to 26.1.0+incompatible
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v26.0.1...v26.1.0)

Updates `github.com/moby/buildkit` from 0.13.1 to 0.13.2
- [Release notes](https://github.com/moby/buildkit/releases)
- [Commits](moby/buildkit@v0.13.1...v0.13.2)

Updates `golang.org/x/sys` from 0.18.0 to 0.19.0
- [Commits](golang/sys@v0.18.0...v0.19.0)

---
updated-dependencies:
- dependency-name: github.com/GoogleContainerTools/kaniko
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/containerd/containerd
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/moby/buildkit
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Bumps the go-dependencies group with 2 updates: [github.com/docker/docker](https://github.com/docker/docker) and [golang.org/x/sys](https://github.com/golang/sys).


Updates `github.com/docker/docker` from 26.1.0+incompatible to 26.1.1+incompatible
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v26.1.0...v26.1.1)

Updates `golang.org/x/sys` from 0.19.0 to 0.20.0
- [Commits](golang/sys@v0.19.0...v0.20.0)

---
updated-dependencies:
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
…t compat (#1352)

* Reorder functions in file

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Also read distro information from /etc/os-release when checking target compat

buildpacks/lifecycle#1347 reads the file when providing target env vars
to buildpacks during detect, but we also need to consider this info when deciding whether or not to run
detect for the buildpack

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Error if we don't find run image OS during analyze

And remove checks for missing OS later in the build, as it should always be there

Signed-off-by: Natalie Arellano <narellano@vmware.com>

---------

Signed-off-by: Natalie Arellano <narellano@vmware.com>
* fix: instance create issue for GH actions

Signed-off-by: Dilip Gowda Bhagavan <dilip.bhagavan@ibm.com>

* fix: ibm cloud instance creation issue

Signed-off-by: Dilip Gowda Bhagavan <dilip.bhagavan@ibm.com>

---------

Signed-off-by: Dilip Gowda Bhagavan <dilip.bhagavan@ibm.com>
* More fixes for target compat checking during detect

- If a buildpack fails to specify os/arch (but specifies distro) still check targets
- If the run image fails to specify os/arch (this should not happen actually as we will fail during analyze) still check targets
- Fix typo in buildpack descriptor struct so that we actually get stack information
- If we get distro information from /etc/os-release, persist this information to later invocations to that the log message
  printed when errors are encountered will be accurate
- Don't override inner `i` in loop (this should not actually affect the outer loop but is confusing)

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* When stack is "any", don't infer empty target as it is not needed

Missing targets is sufficient for wildcard match

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Remove backwards compatible glue that actually causes fewer builds to succeed

Fixes buildpacks/lifecycle#1355

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Remove exit (this was added for debugging purposes)

Signed-off-by: Natalie Arellano <narellano@vmware.com>

---------

Signed-off-by: Natalie Arellano <narellano@vmware.com>
Bumps [azure/docker-login](https://github.com/azure/docker-login) from 1 to 2.
- [Release notes](https://github.com/azure/docker-login/releases)
- [Commits](Azure/docker-login@v1...v2)

---
updated-dependencies:
- dependency-name: azure/docker-login
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Bump the go-dependencies group across 1 directory with 5 updates

Bumps the go-dependencies group with 3 updates in the / directory: [github.com/BurntSushi/toml](https://github.com/BurntSushi/toml), [github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) and [github.com/containerd/containerd](https://github.com/containerd/containerd).


Updates `github.com/BurntSushi/toml` from 1.3.2 to 1.4.0
- [Release notes](https://github.com/BurntSushi/toml/releases)
- [Commits](BurntSushi/toml@v1.3.2...v1.4.0)

Updates `github.com/GoogleContainerTools/kaniko` from 1.22.0 to 1.23.0
- [Release notes](https://github.com/GoogleContainerTools/kaniko/releases)
- [Changelog](https://github.com/GoogleContainerTools/kaniko/blob/main/CHANGELOG.md)
- [Commits](GoogleContainerTools/kaniko@v1.22.0...v1.23.0)

Updates `github.com/awslabs/amazon-ecr-credential-helper/ecr-login` from 0.0.0-20231213181459-b0fcec718dc6 to 0.0.0-20240419161514-af205d85bb44
- [Release notes](https://github.com/awslabs/amazon-ecr-credential-helper/releases)
- [Changelog](https://github.com/awslabs/amazon-ecr-credential-helper/blob/main/CHANGELOG.md)
- [Commits](https://github.com/awslabs/amazon-ecr-credential-helper/commits)

Updates `github.com/containerd/containerd` from 1.7.16 to 1.7.17
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](containerd/containerd@v1.7.16...v1.7.17)

Updates `github.com/docker/docker` from 26.1.1+incompatible to 26.1.2+incompatible
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v26.1.1...v26.1.2)

---
updated-dependencies:
- dependency-name: github.com/BurntSushi/toml
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/GoogleContainerTools/kaniko
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/awslabs/amazon-ecr-credential-helper/ecr-login
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/containerd/containerd
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Fix unit

Signed-off-by: Natalie Arellano <narellano@vmware.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
@joeybrown-sf joeybrown-sf requested a review from a team as a code owner July 2, 2024 20:38
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
pbusko and others added 2 commits July 3, 2024 10:01
* Ensure read access to the run image selected by extensions

Co-authored-by: Nicolas Bender <nicolas.bender@sap.com>
Signed-off-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Pavel Busko <pavel.busko@sap.com>

* move read access check to the restorer cmd

Signed-off-by: Pavel Busko <pavel.busko@sap.com>

* guard behind platform version check

Signed-off-by: Pavel Busko <pavel.busko@sap.com>

---------

Signed-off-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Nicolas Bender <nicolas.bender@sap.com>
* Restore cached launch layers not found in appLayers

Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>
Signed-off-by: Pavel Busko <pavel.busko@sap.com>

* add platform api guard

Signed-off-by: Pavel Busko <pavel.busko@sap.com>

---------

Signed-off-by: Pavel Busko <pavel.busko@sap.com>
Co-authored-by: Ralf Pannemans <ralf.pannemans@sap.com>
phase/restorer.go Outdated Show resolved Hide resolved
phase/cache.go Outdated Show resolved Hide resolved
phase/exporter.go Outdated Show resolved Hide resolved
cache/image_cache.go Outdated Show resolved Hide resolved
cache/image_cache.go Outdated Show resolved Hide resolved
cache/volume_cache.go Outdated Show resolved Hide resolved
cache/volume_cache.go Outdated Show resolved Hide resolved
cache/image_cache.go Show resolved Hide resolved
cache/volume_cache.go Outdated Show resolved Hide resolved
cache/volume_cache.go Outdated Show resolved Hide resolved
@natalieparellano
Copy link
Member

Thank you for this @joeybrown-sf! I left a few nits and a few questions

joeybrown-sf and others added 7 commits July 9, 2024 10:10
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
joeybrown-sf and others added 9 commits July 9, 2024 11:13
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
Signed-off-by: Joey Brown <brown.joseph@salesforce.com>
* Target data: populate os/arch as well as distro information

Fixes buildpacks/lifecycle#1371

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Only os and arch are truly required

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Fix one more unit

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Fix even more units

Signed-off-by: Natalie Arellano <narellano@vmware.com>

---------

Signed-off-by: Natalie Arellano <narellano@vmware.com>
* Bump the go-dependencies group across 1 directory with 6 updates

Bumps the go-dependencies group with 5 updates in the / directory:

| Package | From | To |
| --- | --- | --- |
| [github.com/GoogleContainerTools/kaniko](https://github.com/GoogleContainerTools/kaniko) | `1.23.0` | `1.23.1` |
| [github.com/containerd/containerd](https://github.com/containerd/containerd) | `1.7.17` | `1.7.19` |
| [github.com/google/go-containerregistry](https://github.com/google/go-containerregistry) | `0.19.1` | `0.20.0` |
| [github.com/moby/buildkit](https://github.com/moby/buildkit) | `0.13.2` | `0.14.1` |
| [golang.org/x/sys](https://github.com/golang/sys) | `0.20.0` | `0.22.0` |



Updates `github.com/GoogleContainerTools/kaniko` from 1.23.0 to 1.23.1
- [Release notes](https://github.com/GoogleContainerTools/kaniko/releases)
- [Changelog](https://github.com/GoogleContainerTools/kaniko/blob/main/CHANGELOG.md)
- [Commits](GoogleContainerTools/kaniko@v1.23.0...v1.23.1)

Updates `github.com/containerd/containerd` from 1.7.17 to 1.7.19
- [Release notes](https://github.com/containerd/containerd/releases)
- [Changelog](https://github.com/containerd/containerd/blob/main/RELEASES.md)
- [Commits](containerd/containerd@v1.7.17...v1.7.19)

Updates `github.com/docker/docker` from 26.1.2+incompatible to 26.1.3+incompatible
- [Release notes](https://github.com/docker/docker/releases)
- [Commits](moby/moby@v26.1.2...v26.1.3)

Updates `github.com/google/go-containerregistry` from 0.19.1 to 0.20.0
- [Release notes](https://github.com/google/go-containerregistry/releases)
- [Changelog](https://github.com/google/go-containerregistry/blob/main/.goreleaser.yml)
- [Commits](google/go-containerregistry@v0.19.1...v0.20.0)

Updates `github.com/moby/buildkit` from 0.13.2 to 0.14.1
- [Release notes](https://github.com/moby/buildkit/releases)
- [Commits](moby/buildkit@v0.13.2...v0.14.1)

Updates `golang.org/x/sys` from 0.20.0 to 0.22.0
- [Commits](golang/sys@v0.20.0...v0.22.0)

---
updated-dependencies:
- dependency-name: github.com/GoogleContainerTools/kaniko
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/containerd/containerd
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/docker/docker
  dependency-type: direct:production
  update-type: version-update:semver-patch
  dependency-group: go-dependencies
- dependency-name: github.com/google/go-containerregistry
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: github.com/moby/buildkit
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
- dependency-name: golang.org/x/sys
  dependency-type: direct:production
  update-type: version-update:semver-minor
  dependency-group: go-dependencies
...

Signed-off-by: dependabot[bot] <support@github.com>

* Stay on the same version of buildkit as kaniko

Signed-off-by: Natalie Arellano <narellano@vmware.com>

---------

Signed-off-by: dependabot[bot] <support@github.com>
Signed-off-by: Natalie Arellano <narellano@vmware.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Natalie Arellano <narellano@vmware.com>
Bumps [google.golang.org/grpc](https://github.com/grpc/grpc-go) from 1.64.0 to 1.64.1.
- [Release notes](https://github.com/grpc/grpc-go/releases)
- [Commits](grpc/grpc-go@v1.64.0...v1.64.1)

---
updated-dependencies:
- dependency-name: google.golang.org/grpc
  dependency-type: indirect
...

Signed-off-by: dependabot[bot] <support@github.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
* Surface registry error instead of logging it as debug

When permissions issues are encountered, it can be hard to determine the root cause
without the error returned from the registry

Signed-off-by: Natalie Arellano <narellano@vmware.com>

* Fix test expectation

Signed-off-by: Natalie Arellano <narellano@vmware.com>

---------

Signed-off-by: Natalie Arellano <narellano@vmware.com>
Image utils should fail with a Layer Not found in both ReuseLayer & GetLayer.

For GetLayer, when there is a missing blob, it's return an unexpected EOF error.

For ReuseLayer, when there is a missing blob, it's not returning an error but it should.
@natalieparellano
Copy link
Member

Looks like these units are failing:

=== CONT  TestImageCache/ImageCache/#Commit/with_#ReuseLayer/reuse_without_commit/retrieve_returns_the_previous_layer
    image_cache_test.go:278: Expected nil: failed to reuse cache layer with SHA 'sha256:797bb0abff798d7200af7685dca7901edffc52bf2[650](https://github.com/buildpacks/lifecycle-private/actions/runs/9958919901/job/27514455016?pr=16#step:6:651)0d5bd97282658ee24152'
=== CONT  TestImageCache/ImageCache/#Commit/with_#ReuseLayer/reuse_then_commit/returns_the_reused_layer
    image_cache_test.go:252: Expected nil: failed to reuse cache layer with SHA 'sha256:797bb0abff798d7200af7685dca7901edffc52bf26500d5bd97282658ee24152'

@joeybrown-sf
Copy link
Collaborator Author

oops I introduced a regression. Thanks for that call out!

I think we're good now. Please take a look at your convenience 😃

@natalieparellano
Copy link
Member

Thank you for the PR @joeybrown-sf! This looks good to me.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Security review: recover corrupt caches
5 participants