Fix kanister service port issue in case if webhook is disabled

[viveksinghggits]

Change Overview

In the cases where we install Kanister with webhook enabled we create a service (443) that forwards the request to port 9443 on container. When we upgrade kanister with --set bpValidatingWebhook.enabled=false the service should listen on 8000 and forward the request to port 8000 on container.
This was not happening already, this PR tries to do that.

There was another request in the issue #1376, that says the metrics and healthz endpoint should also not be changed, if we toggle the value bpValidatingWebhook.enabled. Should we handle that as separate PR.

Pull request type

Please check the type of change your PR introduces:

• 🚧 Work in Progress
• 🌈 Refactoring (no functional changes, no api changes)
• 🐹 Trivial/Minor
• 🐛 Bugfix
• 🌻 Feature
• 🗺️ Documentation
• 🤖 Test

Issues

• [BUG] Disabling webhook breaks metrics- and health-endpoints #1376

Test Plan

• 💪 Manual
• ⚡ Unit test
• 💚 E2E

[github-actions]

Thanks for submitting this pull request 🎉. The team will review it soon and get back to you.

If you haven't already, please take a moment to review our project contributing guideline and Code of Conduct document.

[viveksinghggits]

Start with default conf

Install kanister operator (by default webhook would be enabled)

# curl https://kanister-kanister-operator.kanister.svc.cluster.local/v0/healthz -k
{"alive":true,"version":"0.79.0"}

Upgrade the installed release and disable webhook

» helm upgrade kanister helm/kanister-operator -n kanister --set bpValidatingWebhook.enabled=false

# curl kanister-kanister-operator.kanister.svc.cluster.local:8000/v0/healthz
{"alive":true,"version":"0.79.0"}

Upgrade again to enable the validing webhook

» helm upgrade kanister helm/kanister-operator -n kanister --set bpValidatingWebhook.enabled=true

# curl https://kanister-kanister-operator.kanister.svc.cluster.local/v0/healthz -k
{"alive":true,"version":"0.79.0"}

Strart with validating webhook disabled

» helm install kanister helm/kanister-operator -n kanister --set bpValidatingWebhook.enabled=false

# curl kanister-kanister-operator.kanister.svc.cluster.local:8000/v0/healthz
{"alive":true,"version":"0.79.0"}

Upgrade kanister and enable webhook server

# curl https://kanister-kanister-operator.kanister.svc.cluster.local/v0/healthz -k
{"alive":true,"version":"0.79.0"}

[viveksinghggits]

The other thing that I want to talk about is, if we really want to have same endpoint for metrics and healthz irrespective of the fact that validating webhook is enabled or disabled.

[ihcsim]

LGTM 👍. Thanks for fixing this.