Feature: cacheSpec with nodeSelector matching no nodes

[senthilrch]

When the cacheSpec contains an image list with a node selector, kube-fledged caches images onto the nodes matching the selector. If no nodes match the selector, such a situation should be handled:-

• even if during creation the node selector matched some nodes, those nodes might be taken offline during the lifetime of the cluster. during refresh, kube-fledged needs to handle this situation gracefully
• on the other hand, during creation the node selector matched no nodes, it's possible new nodes matching the selector would get added to the cluster. during refresh images should get cached onto these new nodes.

Log of controller when node selector matches no nodes:-

I0801 06:04:22.322180 1 controller.go:633] Completed sync actions for image cache imagecache1(statusupdate)
I0801 06:04:22.322226 1 event.go:282] Event(v1.ObjectReference{Kind:"ImageCache", Namespace:"kubefledged-operator", Name:"imagecache1", UID:"654538c4-8981-47ca-a84d-33c79d877bae", APIVersion:"kubefledged.io/v1alpha2", ResourceVersion:"64822419", FieldPath:""}): type: 'Normal' reason: 'ImageCacheCreate' All requested images pulled succesfully to respective nodes
I0801 06:15:38.801750 1 controller.go:430] Starting to sync image cache imagecache1(create)
E0801 06:15:38.835838 1 controller.go:510] NodeSelector map[tier:backend] did not match any nodes.
E0801 06:15:38.835871 1 controller.go:367] error syncing imagecache: NodeSelector map[tier:backend] did not match any nodes
E0801 06:15:38.835882 1 controller.go:378] error syncing imagecache: NodeSelector map[tier:backend] did not match any nodes
I0801 06:15:38.855125 1 image_manager.go:428] Job imagecache1-2q8wt created (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-25, runtime: containerd://1.4.4)
I0801 06:15:38.855249 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-25, runtime: containerd://1.4.4)
I0801 06:15:38.859396 1 image_manager.go:428] Job imagecache1-ffm5x created (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-26, runtime: containerd://1.4.4)
I0801 06:15:38.859482 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-26, runtime: containerd://1.4.4)
I0801 06:15:38.859525 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0801 06:15:38.859551 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-27, runtime: containerd://1.4.4)
I0801 06:15:38.859588 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-20, runtime: containerd://1.4.4)
I0801 06:15:38.859621 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-20, runtime: containerd://1.4.4)
I0801 06:15:38.863611 1 image_manager.go:428] Job imagecache1-nlpgq created (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0801 06:15:38.863676 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0801 06:15:38.867598 1 image_manager.go:428] Job imagecache1-4sb8q created (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0801 06:15:38.867670 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0801 06:15:38.870791 1 image_manager.go:428] Job imagecache1-ccxvh created (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-23, runtime: containerd://1.4.4)
I0801 06:15:38.870871 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-23, runtime: containerd://1.4.4)
I0801 06:15:38.874277 1 image_manager.go:428] Job imagecache1-flvwd created (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-24, runtime: containerd://1.4.4)
I0801 06:15:38.874333 1 image_manager.go:430] Job not created (image-already-present:- quay.io/bitnami/tomcat:10.0.8 --> node-10-120-127-24, runtime: containerd://1.4.4)
I0801 06:15:40.996335 1 image_manager.go:179] Job imagecache1-2q8wt succeeded (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-25, runtime: containerd://1.4.4)
I0801 06:15:41.096969 1 image_manager.go:179] Job imagecache1-flvwd succeeded (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-24, runtime: containerd://1.4.4)
I0801 06:15:41.409195 1 image_manager.go:179] Job imagecache1-ccxvh succeeded (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-23, runtime: containerd://1.4.4)
I0801 06:15:41.630496 1 image_manager.go:179] Job imagecache1-nlpgq succeeded (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-21, runtime: containerd://1.4.4)
I0801 06:15:41.717570 1 image_manager.go:179] Job imagecache1-4sb8q succeeded (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-22, runtime: containerd://1.4.4)
I0801 06:15:42.435333 1 image_manager.go:179] Job imagecache1-ffm5x succeeded (pull:- quay.io/bitnami/nginx:1.21.1 --> node-10-120-127-26, runtime: containerd://1.4.4)

Output of kubectl get pods showing image-puller pods not deleted:-

NAME                                           READY   STATUS      RESTARTS   AGE
imagecache1-2q8wt-8s68b                        0/1     Completed   0          10m
imagecache1-4sb8q-8nxdn                        0/1     Completed   0          10m
imagecache1-ccxvh-8jcmr                        0/1     Completed   0          10m
imagecache1-ffm5x-nxqwm                        0/1     Completed   0          10m
imagecache1-flvwd-7xfnq                        0/1     Completed   0          10m
imagecache1-nlpgq-r4cts                        0/1     Completed   0          10m
kube-fledged-controller-5c487db996-frgtj       1/1     Running     0          22m
kube-fledged-webhook-server-79f949475c-gkmxb   1/1     Running     0          22m

Output of get imagecache showing status locked in processing:-

apiVersion: kubefledged.io/v1alpha2
kind: ImageCache
metadata:
  annotations:
    kubectl.kubernetes.io/last-applied-configuration: |
      {"apiVersion":"kubefledged.io/v1alpha2","kind":"ImageCache","metadata":{"annotations":{},"labels":{"app":"kubefledged","component":"imagecache"},"name":"imagecache1","namespace":"kube-fledged"},"spec":{"cacheSpec":[{"images":["quay.io/bitnami/nginx:1.21.1","quay.io/bitnami/tomcat:10.0.8"]},{"images":["quay.io/bitnami/redis:6.2.5","quay.io/bitnami/mariadb:10.5.11"],"nodeSelector":{"tier":"backend"}}],"imagePullSecrets":[{"name":"myregistrykey"}]}}
  creationTimestamp: "2021-08-01T06:15:38Z"
  generation: 2
  labels:
    app: kubefledged
    component: imagecache
  managedFields:
  - apiVersion: kubefledged.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:metadata:
        f:annotations:
          .: {}
          f:kubectl.kubernetes.io/last-applied-configuration: {}
        f:labels:
          .: {}
          f:app: {}
          f:component: {}
      f:spec:
        .: {}
        f:cacheSpec: {}
        f:imagePullSecrets: {}
    manager: kubectl-client-side-apply
    operation: Update
    time: "2021-08-01T06:15:38Z"
  - apiVersion: kubefledged.io/v1alpha2
    fieldsType: FieldsV1
    fieldsV1:
      f:status:
        .: {}
        f:message: {}
        f:reason: {}
        f:startTime: {}
        f:status: {}
    manager: kubefledged-controller
    operation: Update
    time: "2021-08-01T06:15:38Z"
  name: imagecache1
  namespace: kube-fledged
  resourceVersion: "64827119"
  selfLink: /apis/kubefledged.io/v1alpha2/namespaces/kube-fledged/imagecaches/imagecache1
  uid: 8117c684-12a7-45f7-b2ba-8ed6975f7f76
spec:
  cacheSpec:
  - images:
    - quay.io/bitnami/nginx:1.21.1
    - quay.io/bitnami/tomcat:10.0.8
  - images:
    - quay.io/bitnami/redis:6.2.5
    - quay.io/bitnami/mariadb:10.5.11
    nodeSelector:
      tier: backend
  imagePullSecrets:
  - name: myregistrykey
status:
  message: Images are being pulled on to the nodes. Please view the status after some
    time
  reason: ImageCacheCreate
  startTime: "2021-08-01T06:15:38Z"
  status: Processing