Mutate eks-prow-build-cluster image pulls to use ECR

[upodroid]

Can we look in to using Kyverno to mutate image pulls on the eks-prow-build-cluster?

It will be faster and cheaper to pull images from ECR than the staging GCR registries.

Good candidates for sync to ECR would would be

• gcr.io/k8s-prow/*
• gcr.io/k8s-staging-test-infra/*
• gcr.io/k8s-staging-*/*

Kyverno would rewrite the image that the submitted from gcr.io/k8s-staging-test-infra/kubekins-e2e:v20230222-b5208facd4-master to ACCOUNT_NUMBER.dkr.ecr.us-east-2.amazonaws.com/[REPO_NAME]/kubekins-e2e:v20230222-b5208facd4-master

We would also have a prowjob that syncs that ECR repo with images from the staging repos.

Sample Kyverno policy:

apiVersion: kyverno.io/v1
kind: ClusterPolicy
metadata:
  name: kubernetes-gcr-to-registry    
spec:
  background: false
  rules:
    - name: replace-image-registry-pod-containers
      match:
        any:
        - resources:
            kinds:
            - Pod
      mutate:
        foreach:
        - list: "request.object.spec.containers"
          patchStrategicMerge:
            spec:
              containers:
              - name: "{{ element.name }}"
                image: "{{ replace_all('{{element.image}}', 'k8s.gcr.io', 'registry.k8s.io' )}}"
    - name: replace-image-registry-pod-initcontainers
      match:
        any:
        - resources:
            kinds:
            - Pod
      preconditions:
        all:
        - key: "{{ request.object.spec.initContainers[] || `[]` | length(@) }}"
          operator: GreaterThanOrEquals
          value: 1
      mutate:
        foreach:
        - list: "request.object.spec.initContainers"
          patchStrategicMerge:
            spec:
              initContainers:
              - name: "{{ element.name }}"
                image: "{{ replace_all('{{element.image}}', 'k8s.gcr.io', 'registry.k8s.io' )}}"

/cc @ameukam @xmudrii

[sftim]

Is ECR pull-through cache useful here?

I'm thinking we might need to switch to specifying hashes rather than tags to benefit. Not sure, haven't thought this through.

[BenTheElder]

Is ECR pull-through cache useful here?

I think this came up previously, but lost the thread. Unfortunately it's not because:

Amazon ECR currently supports creating pull through cache rules for Amazon ECR Public and Quay.

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale

ECR now supports registry.k8s.io, we should revisit this.

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[xmudrii]

/remove-lifecycle stale

[xmudrii]

/unassign
/lifecycle frozen