push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error #2423

luxurine · 2023-03-08T14:58:45Z

Actual behavior
When push builded image, with base image(ubuntu:20.04) cached by warmer, will get error:

error pushing image: failed to push to destination luxurine/counter:latest: PUT https://index.docker.io/v2/luxurine/counter/manifests/latest: 
MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' 
not 'application/vnd.oci.image.manifest.v1+json'

(If there is no cache for base image, image push is ok)

Expected behavior
Image should be pushed successfully.

Preliminary analysis

It looks like ubuntu:20.04 use "mediaType": "application/vnd.oci.image.manifest.v1+json", but dockerhub expect application/vnd.docker.distribution.manifest.v2+json.

The strange thing is, if build without base image cache, pushing image is just ok, I guess reasons could be:

if build without base image cache, kaniko will do automatic transformation
if build with base image cache, as cache dir is readonly in kaniko executor's view, it will build on top of the cache

kaniko can cache images in a local directory that can be volume mounted into the kaniko pod. To do so, the cache must first be populated, as it is read-only. We provide a kaniko cache warming image at gcr.io/kaniko-project/warmer

Some proposal：

warmer add flag to control transformation, like OCI --> Docker
executor support base image cache（assume cache dir is readwrite）（related issue Feature request: Cache base image in repository #2257 ）
or, executor copy cached base image and do some transformation before build

To Reproduce
Steps to reproduce the behavior:

prepare manifests

λ tree .
.
├── Dockerfile
├── bin
│   └── counter
└── config.json

2 directories, 5 files

warm base image

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:v1.9.1 --cache-dir=/workspace/cache --image="ubuntu:20.04"
INFO[0000] Retrieving image manifest ubuntu:20.04
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io

build and push image

$ docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.9.1-debug --cache=true --cache-dir=/workspace/cache --dockerfile=Dockerfile --destination=luxurine/counter:latest
INFO[0002] Retrieving image manifest ubuntu:20.04
INFO[0002] Retrieving image ubuntu:20.04 from registry index.docker.io
INFO[0005] Found sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d in local cache
INFO[0005] Found manifest at /workspace/cache/sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d.json
INFO[0005] Built cross stage deps: map[]
INFO[0005] Retrieving image manifest ubuntu:20.04
INFO[0005] Returning cached image manifest
INFO[0005] Found sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d in local cache
INFO[0005] Found manifest at /workspace/cache/sha256:3626dff0d616e8ee7065a9ac8c7117e904a4178725385910eeecd7f1872fc12d.json
INFO[0005] Executing 0 build triggers
INFO[0005] Building stage 'ubuntu:20.04' [idx: '0', base-idx: '-1']
INFO[0005] Unpacking rootfs as cmd COPY bin/ /app requires it.
INFO[0007] COPY bin/ /app
INFO[0007] Taking snapshot of files...
INFO[0007] WORKDIR /app
INFO[0007] Cmd: workdir
INFO[0007] Changed working directory to /app
INFO[0007] No files changed in this command, skipping snapshotting.
INFO[0007] ENTRYPOINT ["/app/counter"]
INFO[0007] No files changed in this command, skipping snapshotting.
INFO[0007] Pushing image to luxurine/counter:latest
error pushing image: failed to push to destination luxurine/counter:latest: PUT https://index.docker.io/v2/luxurine/counter/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

Additional Information

Dockerfile

FROM ubuntu:20.04

COPY bin/ /app

WORKDIR /app

ENTRYPOINT ["/app/counter"]

Kaniko Image: gcr.io/kaniko-project/executor:v1.9.1-debug

ref:

The text was updated successfully, but these errors were encountered:

jamgregory · 2023-03-31T11:27:54Z

I've just experienced a similar issue building a local development image and pushing it to our own GitLab Registry instance (albeit without the useful error message - I just got MANIFEST_INVALID: manifest invalid; map[]

I'm not sure how you worked around this @luxurine, but my solution was to build the Ubuntu base image myself:

# Based on: https://git.launchpad.net/cloud-images/+oci/ubuntu-base/tree/?h=focal-20.04
FROM scratch
ADD source/ubuntu-focal-oci-amd64-root.tar.gz /

SeWieland · 2023-04-21T09:25:58Z

I think this is related to #1836

aaron-prindle · 2023-09-06T17:58:40Z

I believe this is a dupe of #1836 which was recently fixed and in the latest kaniko release (v1.15.0). Closing, feel free to re-open if there is more to address here

luxurine · 2023-10-27T08:04:45Z

I've just tried kaniko-project/executor:v1.17.0 again, still the same problem. Could you please reopen this issue? @aaron-prindle

manifest

# Dockfile
FROM ubuntu:20.04 # with OCI format
WORKDIR /app

step

warmup image

$ docker run -v $(pwd):/workspace gcr.io/kaniko-project/warmer:v1.17.0 --cache-dir=/workspace/cache --image=ubuntu:20.04

# output
INFO[0000] Retrieving image manifest ubuntu:20.04
INFO[0000] Retrieving image ubuntu:20.04 from registry index.docker.io

build and push

$ docker run -ti --rm -v `pwd`:/workspace -v `pwd`/config.json:/kaniko/.docker/config.json:ro gcr.io/kaniko-project/executor:v1.17.0-debug --cache=true --cache-dir=/workspace/cache --dockerfile=Dockerfile --destination=luxurine/demo:latest --verbosity=debug

# output
INFO[0002] Retrieving image manifest ubuntu:20.04
INFO[0002] Retrieving image ubuntu:20.04 from registry index.docker.io
INFO[0006] Found sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d in local cache
INFO[0006] Found manifest at /workspace/cache/sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d.json
INFO[0006] Built cross stage deps: map[]
INFO[0006] Retrieving image manifest ubuntu:20.04
INFO[0006] Returning cached image manifest
INFO[0006] Found sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d in local cache
INFO[0006] Found manifest at /workspace/cache/sha256:218bb51abbd1864df8be26166f847547b3851a89999ca7bfceb85ca9b5d2e95d.json
INFO[0006] Executing 0 build triggers
...
INFO[0006] Pushing image to luxurine/demo:latest
error pushing image: failed to push to destination luxurine/demo:latest: PUT https://index.docker.io/v2/luxurine/demo/manifests/latest: MANIFEST_INVALID: manifest invalid; mediaType in manifest should be 'application/vnd.docker.distribution.manifest.v2+json' not 'application/vnd.oci.image.manifest.v1+json'

tanadeau · 2023-12-07T17:58:26Z

I've also seen this with 1.18.0 and 1.19.0. Seems possibly related to newer issue #2883. We're using GitLab Container Registry and not AWS ECR.

hojo0590 · 2024-02-26T17:44:48Z

maybe related/a regression of #2713?

mhio · 2024-07-16T02:17:49Z

Ran into this issue on executor-v1.23.2 and recent warmer-latest (doesn't report a version) to a gitlab v16.11.5 container registry.

luxurine · 2024-07-16T03:47:07Z

Ran into this issue on executor-v1.23.2 and recent warmer-latest (doesn't report a version) to a gitlab v16.11.5 container registry.

@mhio here is a temporary fix you can try:

fork kaniko repo
update func func (c *cachedImage) Manifest() (*v1.Manifest, error) in pkg/cache/cache.go and return tabball's manifest by default.
build image

Warning

target image is always in docker format

func (c *cachedImage) Manifest() (*v1.Manifest, error) {
	if c.mfst == nil {
		return c.Image.Manifest()
	}
	return c.mfst, nil
}

->

func (c *cachedImage) Manifest() (*v1.Manifest, error) {
	mfst, err := c.Image.Manifest()
	if mfst != nil {
		return mfst, err
	}
	return c.mfst, nil
}

mhio · 2024-07-31T05:16:39Z

Thanks @luxurine we'll give that a try, disabling the warmed cache was the quick workaround.

luxurine closed this as completed May 15, 2023

luxurine changed the title ~~push image to Dockerhub build with cached base image(OCI format) got MANIFEST_INVALID error~~ push image to Dockerhub build with warmed base image(OCI format) got MANIFEST_INVALID error Jul 6, 2023

luxurine reopened this Jul 6, 2023

aaron-prindle closed this as completed Sep 6, 2023

luxurine changed the title ~~push image to Dockerhub build with warmed base image(OCI format) got MANIFEST_INVALID error~~ push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error Oct 27, 2023

aaron-prindle reopened this Dec 7, 2023

QuanZhang-William added feat/warmer kind/bug Something isn't working priority/p1 Basic need feature compatibility with docker build. we should be working on this next. labels Mar 15, 2024

This was referenced Jul 2, 2024

feat: add func tarball.OCIImageFromPath to return v1.Image with OCI mediaType google/go-containerregistry#1969

Open

fix: mismatched mediaType error when push image build with warmed bas… #3231

Draft

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error #2423

push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error #2423

luxurine commented Mar 8, 2023 •

edited

Loading

jamgregory commented Mar 31, 2023

SeWieland commented Apr 21, 2023

aaron-prindle commented Sep 6, 2023

luxurine commented Oct 27, 2023

tanadeau commented Dec 7, 2023

hojo0590 commented Feb 26, 2024

mhio commented Jul 16, 2024

luxurine commented Jul 16, 2024 •

edited

Loading

mhio commented Jul 31, 2024

push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error #2423

push image (with warmed base image(OCI format)) to Dockerhub got MANIFEST_INVALID error #2423

Comments

luxurine commented Mar 8, 2023 • edited Loading

jamgregory commented Mar 31, 2023

SeWieland commented Apr 21, 2023

aaron-prindle commented Sep 6, 2023

luxurine commented Oct 27, 2023

tanadeau commented Dec 7, 2023

hojo0590 commented Feb 26, 2024

mhio commented Jul 16, 2024

luxurine commented Jul 16, 2024 • edited Loading

mhio commented Jul 31, 2024

luxurine commented Mar 8, 2023 •

edited

Loading

luxurine commented Jul 16, 2024 •

edited

Loading