Finalizers permissions missing in Openshift

[rollandf]

When deployed in Openshift, finalizers permissions are required to be able to set ownerReference.
We need to add these permissions on all objects we want to add an ownerReference.

Errors example:

2023-10-16T06:52:21Z	ERROR	Error while syncing state	{"controller": "nicclusterpolicy", "controllerGroup": "mellanox.com", "controllerKind": "NicClusterPolicy", "NicClusterPolicy": {"name":"nic-cluster-policy"}, "namespace": "", "name": "nic-cluster-policy", "reconcileID": "855ed7eb-3273-4aaf-90ab-2dc60e11a678", "error": "failed to create/update objects: configmaps \"sriovdp-config\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>", "errorVerbose": "configmaps \"sriovdp-config\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nfailed to create/update objects\ngit.luolix.top/Mellanox/network-operator/pkg/state.(*stateSriovDp).Sync\n\t/workspace/pkg/state/state_sriov_dp.go:116\ngit.luolix.top/Mellanox/network-operator/pkg/state.(*stateManager).SyncState\n\t/workspace/pkg/state/manager.go:92\ngit.luolix.top/Mellanox/network-operator/controllers.(*NicClusterPolicyReconciler).Reconcile\n\t/workspace/controllers/nicclusterpolicy_controller.go:140\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
github.com/Mellanox/network-operator/pkg/state.(*stateManager).SyncState
	/workspace/pkg/state/manager.go:101
github.com/Mellanox/network-operator/controllers.(*NicClusterPolicyReconciler).Reconcile
	/workspace/controllers/nicclusterpolicy_controller.go:140
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

2023-10-16T08:13:16Z    ERROR   Error while syncing state       {"controller": "nicclusterpolicy", "controllerGroup": "mellanox.com", "controllerK
ind": "NicClusterPolicy", "NicClusterPolicy": {"name":"nic-cluster-policy"}, "namespace": "", "name": "nic-cluster-policy", "reconcileID": "19ddc9
07-ecf9-47f8-8dc1-37e8728d5326", "error": "failed to create/update objects: customresourcedefinitions.apiextensions.k8s.io \"ippools.nv-ipam.nvidi
a.com\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>", "errorVerbose
": "customresourcedefinitions.apiextensions.k8s.io \"ippools.nv-ipam.nvidia.com\" is forbidden: cannot set blockOwnerDeletion if an ownerReference
 refers to a resource you can't set finalizers on: , <nil>\nfailed to create/update objects\ngit.luolix.top/Mellanox/network-operator/pkg/state.(*stat
eNVIPAMCNI).Sync\n\t/workspace/pkg/state/state_nv_ipam_cni.go:115\ngit.luolix.top/Mellanox/network-operator/pkg/state.(*stateManager).SyncState\n\t/wo
rkspace/pkg/state/manager.go:92\ngit.luolix.top/Mellanox/network-operator/controllers.(*NicClusterPolicyReconciler).Reconcile\n\t/workspace/controller
s/nicclusterpolicy_controller.go:140\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile\n\t/go/pkg/mod/sigs.k8s.io/co
ntroller-runtime@v0.14.6/pkg/internal/controller/controller.go:122\nsigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcile
Handler\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323\nsigs.k8s.io/controller-runtime/pkg/intern
al/controller.(*Controller).processNextWorkItem\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274\ns
igs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2\n\t/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/intern
al/controller/controller.go:235\nruntime.goexit\n\t/usr/local/go/src/runtime/asm_amd64.s:1598"}
github.com/Mellanox/network-operator/pkg/state.(*stateManager).SyncState
        /workspace/pkg/state/manager.go:101
github.com/Mellanox/network-operator/controllers.(*NicClusterPolicyReconciler).Reconcile
        /workspace/controllers/nicclusterpolicy_controller.go:140
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Reconcile
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:122
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).reconcileHandler
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:323
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).processNextWorkItem
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:274
sigs.k8s.io/controller-runtime/pkg/internal/controller.(*Controller).Start.func2.2
        /go/pkg/mod/sigs.k8s.io/controller-runtime@v0.14.6/pkg/internal/controller/controller.go:235

[rollandf]

2023-10-16T08:13:16Z    ERROR   Error while syncing state       {"controller": "nicclusterpolicy", "controllerGroup": "mellanox.com", "controllerK
ind": "NicClusterPolicy", "NicClusterPolicy": {"name":"nic-cluster-policy"}, "namespace": "", "name": "nic-cluster-policy", "reconcileID": "19ddc9
07-ecf9-47f8-8dc1-37e8728d5326", "error": "failed to create/update objects: customresourcedefinitions.apiextensions.k8s.io \"ippools.nv-ipam.nvidi
a.com\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>", "errorVerbose
": "customresourcedefinitions.apiextensions.k8s.io \"ippools.nv-ipam.nvidia.com\" is forbidden: cannot set blockOwnerDeletion if an ownerReference
 refers to a resource you can't set finalizers on: , <nil>\nfailed to create/update objects

2023-10-16T06:52:21Z	ERROR	Error while syncing state	{"controller": "nicclusterpolicy", "controllerGroup": "mellanox.com", "controllerKind": "NicClusterPolicy", "NicClusterPolicy": {"name":"nic-cluster-policy"}, "namespace": "", "name": "nic-cluster-policy", "reconcileID": "855ed7eb-3273-4aaf-90ab-2dc60e11a678", "error": "failed to create/update objects: configmaps \"sriovdp-config\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>", "errorVerbose": "configmaps \"sriovdp-config\" is forbidden: cannot set blockOwnerDeletion if an ownerReference refers to a resource you can't set finalizers on: , <nil>\nfailed to create/update objects\ngit.luolix.top/Mellanox/network-operator/pkg/state.(*stateSriovDp).Sync\n\t/workspace/pkg/state/state_sriov_dp.go:116

[rollandf]

need to add finalizer permission to NICCLusterpolicy

[rollandf]

Fixed in #637