NginX Ingress Controller - provided digest did not match uploaded content

[FrankWigger]

Environment

Device and OS: Debian bookworm
zarf version: v0.32.5
Kubernetes distro being used: from zarf
Other:

Steps to reproduce

1. Create a zarf.yaml withn ginx ingress controller.
2. Call zarf prepare find-images
3. zarf package create.
4. On target maschine call zarf package deplpy

Expected result

Nginx ingress is up and running

Actual Result

Zarf reports
Retrying (1/3) in 5s: PUT
http://127.0.0.1:34819/v2/ingress-nginx/controller/manifests/sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c:
DIGEST_INVALID: provided digest did not match uploaded content

Zarf.yaml

My zarf.yaml looks like:

kind: ZarfPackageConfig # ZarfPackageConfig is the package kind for most normal zarf packages
metadata:
  name: ingress-nginx      
  version: v0.1.2  
  description: |        
    "A Zarf Package that deploys the nginx ingress controller"
components:
  - name: ingress-nginx
    description: |   
      "Deploys nginx ingress controller"
    required: true   
    charts:
      - name: ingress-nginx
        url: https://kubernetes.github.io/ingress-nginx
        version: 4.10.0
        namespace: ingress-nginx
    images:
      - registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
      - registry.k8s.io/ingress-nginx/kube-webhook-certgen:v1.4.0@sha256:44d1d0e9f19c63f58b380c5fddaca7cf22c7cee564adeff365225a5df5ef3334
      # Cosign artifacts for images - ingress-nginx - ingress-nginx
      - registry.k8s.io/ingress-nginx/controller:sha256-42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c.sig
      - registry.k8s.io/ingress-nginx/kube-webhook-certgen:sha256-44d1d0e9f19c63f58b380c5fddaca7cf22c7cee564adeff365225a5df5ef3334.sig

Any ideas?

[AustinAbro321]

Yeah there is currently a bug which is breaking pulling sha's by the index.json. We being work on the fix.

In the meantime, you can either remove the sha and just use a tag or use a sha for the platform you are building your zarf package for. For example, you could run the following command docker buildx imagetools inspect registry.k8s.io/ingress-nginx/controller:v1.10.0 will give the output

Name:      registry.k8s.io/ingress-nginx/controller:v1.10.0
MediaType: application/vnd.docker.distribution.manifest.list.v2+json
Digest:    sha256:42b3f0e5d0846876b1791cd3afeb5f1cbbe4259d6f35651dcc1b5c980925379c
           
Manifests: 
  Name:      registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:67f647d99b087bebd5e8c7792cc7d5dd715343354698c43ea0edab81896b75f0
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/amd64
             
  Name:      registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:8ba60866fa132a2a6cef840f0c6f412b615f0f161455a1771f1e7432eaff1685
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm/v7
             
  Name:      registry.k8s.io/ingress-nginx/controller:v1.10.0@sha256:2e53c57c81ebad0263e98c98b66f0151217ce417eeeaacf911e62cc3dbae27e4
  MediaType: application/vnd.docker.distribution.manifest.v2+json
  Platform:  linux/arm64

You can then add the sha of the platform, you are building the package for, usually amd64 or arm64.

[AustinAbro321]

In the latest release of Zarf we have decided to explicitly not support index image shas on create form the PR #2429. More context is on #2425, but TLDR moving an index image sha into the air gap would require bringing in all the pre-req images defined in that index sha which can be upwards of 10-20.