[BUG] Can't apply blueprint cause webhook can't be reached

[muffl0n]

Describe the bug
When I try to apply a blueprint, I get this error message

Error from server (InternalError): error when creating "blueprint.yaml": Internal error occurred: failed calling webhook "blueprints.cr.kanister.io": failed to call webhook: Post "https://kanister-kanister-operator.kanister.svc:443/validate/v1alpha1/blueprint?timeout=5s": context deadline exceeded

To Reproduce

1. Create blueprint.yaml
2. Call kubectl apply -f blueprint.yaml
3. Get error message

Expected behavior
Blueprint should be applied without an error.

Environment
Kubernetes Version/Provider: GKE v1.22.6-gke.300
Helm-Chart: 0.74.0
Kanister operator: Develop version of PR #1282

Though I'm using the develop version of said PR, I can also reproduce this behaviour with released version ghcr.io/kanisterio/controller:0.74.0

[viveksinghggits]

Hi @muffl0n,
can you please confirm if you are facing this continuously or this is intermittent.

[muffl0n]

I can reproduce this on every try.

[ihcsim]

Are there any relevant logs in the operator?

[muffl0n]

None I found so far. Is there any method to pump up the logging?

[ihcsim]

Should be able to use the LOG_LEVEL env var to set the log level. See supported values here.

[ihcsim]

I see what the issue is - you have to add additional firewall rules to allow traffic between your private GKE apiserver and the Kanister webhooks. See https://cloud.google.com/kubernetes-engine/docs/how-to/private-clusters#add_firewall_rules.

[muffl0n]

You are right! Thank you very much!

I recall having a similar problem with "Config Connector": GoogleCloudPlatform/k8s-config-connector#423
There, they rolled back a change and used 443 again, instead of 9443. Cause 443 (like 10250) is allowed by default, it always works. See also https://cloud.google.com/kubernetes-engine/docs/release-notes#known-20210222

Is there a special reason why Kanister uses 9443 and not 443? Notice: It's the port of the pod/endpoint, not the service's one.

Edit: Default changed to 9443 some time ago: kubernetes-sigs/controller-runtime#1076
Wonder what's behind that decision.

[ihcsim]

9443 is the default port used in the webhook package. Also, it might have something to do with getting the HTTP server to listen at a lower port number requires the NET_ADMIN capability.

[muffl0n]

Thanks for the clarification! I added a section to "Troubleshooting" in #1309