Allow disabling ControlPlane's admission webhook deployment and management

[pmalek]

Problem statement

https://github.com/Kong/gateway-operator-archive/pull/1545 introduced deploying ControlPlane's (KIC's) admission webhook in an unconditional manner: each ControlPlane gets its admission webhook configuration (and related resources) deployed.

This can be an issue for several use cases:

• KIC's HTTPRoute admission rules do not allow admission of routes' that contain invalid/unsupported Group and/or Kind in spec.rules.backendRefs: https://github.com/Kong/kubernetes-ingress-controller/blob/a9749c8a2589d3b3e421ffe1b5a6ac4d43359c78/internal/admission/validation/gateway/httproute.go#L163-L171.

  • A decision has been made to retain that behavior but to allow users to disable admission webhook (this is already possible in KIC itself) to be conformant with Gateway API (which requires the routes to be admitted and their statuses appropriately filled).

• Due to the nature of KGO generating patches and applying them to resources in the cluster, when the environment mutates resources managed by KGO then it might end up in an endless loop as in AKS' Admission Enforcer mutates ControlPlane's ValidatingWebhookConfiguration which causes a perpetual reconciliation loop #239.

  • Allowing users to disable the creation and management of ControlPlane's admission webhook configuration we'll allow users have a working KGO setup until AKS' Admission Enforcer mutates ControlPlane's ValidatingWebhookConfiguration which causes a perpetual reconciliation loop #239's underlying issue (the way KGO calculates patches and the expected state) is resolved.

Proposed solution

• Detect if ControlPlane has CONTROLLER_ADMISSION_WEBHOOK_LISTEN overridden with off (https://github.com/Kong/kubernetes-ingress-controller/blob/bcdd40a4099b797e274ba72e276a62fc1785053b/internal/manager/setup.go#L201) and if that's the case, disable creation of admission webhook configuration (and other related resources) for ControlPlane.
  • This would not introduce a new field to not allow conflicts with the aforementioned env
• Detect change in CONTROLLER_ADMISSION_WEBHOOK_LISTEN env when ControlPlane is deployed
  • If its value changes from off to anything else (that's a valid <address>:<port> string) then deploy admission webhook configuration
  • If it changes from anything to off then remove admission webhook configuration and any other resources related to it.

Special notes

Putting tentatively in 1.3