Permission issue when purging images from cache on OpenShift

[hfwen0502]

The errors shown in the ImageCache's object:

status:
  completionTime: "2022-02-17T21:03:39Z"
  failures:
    uk.icr.io/hf-roks-uk/in-cluster:latest:
    - message: |
        time="2022-02-17T21:03:37Z" level=fatal msg="connect: connect endpoint 'unix:///var/run/crio/crio.sock', make sure you are running as root and the endpoint has been started: context deadline exceeded"
      node: 10.242.0.18
      reason: Error
  message: Image deletion failed for some images. Please see "failures" section
  reason: ImageCachePurge
  startTime: "2022-02-17T21:03:33Z"
  status: Failed

I can reproduce this by just running this job, rsh to the pod and run the crictl command.

apiVersion: batch/v1
kind: Job
metadata:
  name: hf-job
spec:
  template:
    spec:
      name: hf-pod
      containers:
      - name: c
        image: senthilrch/kubefledged-cri-client:v0.9.0
        command: ["tail", "-f", "/dev/null"]
        securityContext:
          runAsUser: 0
          privileged: true
        volumeMounts:
        - mountPath: /host
          name: host
      hostNetwork: true
      volumes:
      - hostPath:
          path: /
          type: Directory
        name: host

      restartPolicy: Never
      imagePullSecrets:
        - name: all-icr-io

As you can see, even though adding the additional privilege, the container still fails to do "crictl images" or "/usr/bin/crictl --runtime-endpoint=unix:///var/run/crio/crio.sock --image-endpoint=unix:///var/run/crio/crio.sock image".

Any idea if there is a way to resolve this?

[hfwen0502]

Ok. There is a solution to this. Just need to change the file below. With that and adjusting SCC, things would work fine in OpenShift 4.x.

[root@hf-eu-login kube-fledged]# git diff pkg/images/image_helpers.go
diff --git a/pkg/images/image_helpers.go b/pkg/images/image_helpers.go
index dca5a67..778e4a6 100644
--- a/pkg/images/image_helpers.go
+++ b/pkg/images/image_helpers.go
@@ -205,6 +205,7 @@ func newImageDeleteJob(imagecache *fledgedv1alpha2.ImageCache, image string, nod
                                                        },
                                                },
                                        },
+                                       HostNetwork:      true,
                                        RestartPolicy:    corev1.RestartPolicyNever,
                                        ImagePullSecrets: imagecache.Spec.ImagePullSecrets,
                                        Tolerations: []corev1.Toleration{

[senthilrch]

@hfwen0502 : Thanks for analysing the issue.

I propose: A new flag is needed in kubefledged-controller --image-delete-job-host-network. The value of this new flag should be passed to Image manager routine when it is started. func newImageDeleteJob should set "HostNetwork: true" when --image-delete-job-host-network=true.

Changes needed in helm chart as well.

[senthilrch]

SODACODE22: Raise PR against "develop" branch.

[hfwen0502]

That's fantastic. Thank you!

[bhuvanessr]

Pls. assign this issue to me..I'll try it...many changes needed. it should be SODAHIGH??