Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Default image output from buildx v0.10 cannot run on Google Cloud Run or AWS Lambda #1533

Closed
jedevc opened this issue Jan 20, 2023 · 28 comments

Comments

@jedevc
Copy link
Collaborator

jedevc commented Jan 20, 2023

As of Buildx 0.10, images are built with provenance -- this behavior is opt-out, and can be disabled with --provenance=false if using the buildx build cli, or provenance: false if using docker/build-push-action.

Images built with these default attestations that use the attestation storage from buildkit cannot be run on Google Cloud Run or AWS Lambda. While both Cloud Run and Lambda support the OCI format, they do not support multi-platform images.

From Google Cloud Run docs (emphasis mine):

Cloud Run accepts container images in the Docker Image Manifest V2, Schema 1, Schema 2, and OCI image formats.

Manifest lists used for Multi-Architecture Images are not supported.

Note

Cloud Run has now added support for processing image indexes: #1533 (comment).

From Lambda docs (emphasis mine):

Lambda provides multi-architecture base images. However, the image you build for your function must target only one of the architectures. Lambda does not support functions that use multi-architecture container images.

Ideally, GCR and Lambda should support multi-platform images, and detect the current platform from the Docker manifest list / OCI index.

As a temporary workaround (in order of preference):

  • Users should set --provenance=false on buildx build, or set provenance: false on docker/build-push-action
  • Users can force an explicit buildx version to v0.9.1
  • Users can force an explicit buildkit version to v0.10
peteryates added a commit to DFE-Digital/early-careers-framework that referenced this issue Jan 20, 2023
Builds were failing with the error:

> OCI index found, but Accept header does not support OCI indexes

The suggested fix is to temporarily disable provenance until better
support for OCI indexes arrives.

docker/buildx#1533
peteryates added a commit to DFE-Digital/early-careers-framework that referenced this issue Jan 20, 2023
Builds were failing with the error:

> OCI index found, but Accept header does not support OCI indexes

The suggested fix is to temporarily disable provenance until better
support for OCI indexes arrives.

docker/buildx#1533
peteryates added a commit to DFE-Digital/early-careers-framework that referenced this issue Jan 20, 2023
Builds were failing with the error:

> OCI index found, but Accept header does not support OCI indexes

The suggested fix is to temporarily disable provenance until better
support for OCI indexes arrives.

docker/buildx#1533
leandroalemao pushed a commit to DFE-Digital/early-careers-framework that referenced this issue Jan 20, 2023
Builds were failing with the error:

> OCI index found, but Accept header does not support OCI indexes

The suggested fix is to temporarily disable provenance until better
support for OCI indexes arrives.

docker/buildx#1533
leandroalemao pushed a commit to DFE-Digital/early-careers-framework that referenced this issue Jan 20, 2023
Builds were failing with the error:

> OCI index found, but Accept header does not support OCI indexes

The suggested fix is to temporarily disable provenance until better
support for OCI indexes arrives.

docker/buildx#1533
leandroalemao pushed a commit to DFE-Digital/early-careers-framework that referenced this issue Jan 20, 2023
Builds were failing with the error:

> OCI index found, but Accept header does not support OCI indexes

The suggested fix is to temporarily disable provenance until better
support for OCI indexes arrives.

docker/buildx#1533
@steren
Copy link

steren commented Jan 20, 2023

Cloud Run PM here, we received many reports about this. At the moment, Cloud Run doesn't support multi architecture images.

We were tracking this work item, but we do not have an delivery date to communicate yet.

JonnyWong16 added a commit to Tautulli/Tautulli that referenced this issue Jan 21, 2023
jgarber623 added a commit to jgarber623/indieweb-endpoints.cc that referenced this issue Jan 21, 2023
You gotta be kidding me.

See: docker/buildx#1533
jgarber623 added a commit to jgarber623/micromicro.cc that referenced this issue Jan 21, 2023
You gotta be kidding me.

See: docker/buildx#1533
jgarber623 added a commit to jgarber623/rel-me.cc that referenced this issue Jan 21, 2023
You gotta be kidding me.

See: docker/buildx#1533
@piotrekkr
Copy link

@steren If you are Cloud Run PM then I have a request that can help save few hours of debugging for everyone. Would be nice to return proper error message when we try to use multi-platform images with cloud run instead of returning

Image 'europe-docker.pkg.dev/xxx/xxxx' not found

It clearly exist in artifact registry since I choose it from select list when creating cloud run service.

Message like:

Image 'europe-docker.pkg.dev/xxx/xxxx'  is a multi-platform image and is not supported

Would probably reduce debug time to 15 min instead of 6h.

Thanks

@davideme
Copy link

davideme commented Jan 23, 2023

@steren and buildx team, Cloud Run supports SLSA Build level 3 through provenance, see https://cloud.google.com/software-supply-chain-security/docs/sds/deploy-run-view-security-insights#build
Is there a way to support provenance in buildx without multi architecture images?
Or the way provenance is added by Google Cloud Build and buildx is not the same?

@billinghamj
Copy link

Cloud Run was supporting the old Docker "fat manifest" manifest lists application/vnd.docker.distribution.manifest.list.v2+json - we've been using them for over a year

It just doesn't support the new OCI format

@billinghamj
Copy link

I would also note that the Docker documentation seemingly isn't up to date on covering this change: https://docs.docker.com/registry/spec/manifest-v2-2/

JosephKav added a commit to release-argus/Argus that referenced this issue Jul 19, 2023
ghcr/quay have been giving unknown/unknown images since I released 0.12.0

docker/build-push-action#900 (comment)
JosephKav added a commit to release-argus/Argus that referenced this issue Jul 19, 2023
ghcr/quay have been giving unknown/unknown images since I released 0.12.0

docker/build-push-action#900 (comment)
keithly added a commit to keithly/lambda-python-custom that referenced this issue Nov 1, 2023
keithly added a commit to keithly/lambda-python-custom that referenced this issue Nov 1, 2023
mcdonnnj added a commit to cisagov/skeleton-docker that referenced this issue Dec 6, 2023
There is currently a potential incompatibility with the default
behavior of the version of buildx being used. A default image generated
is built with provenance, which is something we would like to have, but
these default images can run on neither Google Cloud Run nor AWS
Lambda. Please see docker/buildx#1533 for mroe information. Since we
want to retain support for creating AWS Lambda images we add a
commented out disabling of this functionality that can be enabled in a
downstream repository if needed.
steflsd added a commit to WeR1Hub/.github that referenced this issue Dec 8, 2023
@majidakbaridh
Copy link

Is there any way to use newer version of setup-buildx-action but changing mediaType to "mediaType": "application/vnd.docker.distribution.manifest.list.v2+json" ?

@tonistiigi
Copy link
Member

@majidakbaridh you can use -o oci-mediatypes=false but if you want to avoid attestations then disable attestations directly with --provenance=false. You can't have the attestations without the OCI mediatypes as well.

gabeio added a commit to earnnest-greenville/docker-kong that referenced this issue Feb 21, 2024
docker/buildx#1533

our ecr repository has multiple shards of different images and it's hard
to tell when vulnerability scans are accurate or not when the images are
in multiple pieces like they are now.

also bumps other action versions to more current versions
jgarber623 added a commit to jgarber623/indieweb-endpoints.cc that referenced this issue May 17, 2024
Artifact Registry may support this. See:

docker/buildx#1533 (comment)

This reverts commit 216249d.
thehajime added a commit to ukontainer/rumprun-packages that referenced this issue Jul 12, 2024
docker/buildx#1533

Signed-off-by: Hajime Tazaki <thehajime@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests