TestWebhook failure

[thbkrkr]

Since

• Update module sigs.k8s.io/controller-runtime to v0.15.0 #6847

The e2e test TestWebhook is failing with this stacktrace:

2023/06/02 11:10:28 http: panic serving [::1]:62541: runtime error: invalid memory address or nil pointer dereference
goroutine 1362 [running]:
net/http.(*conn).serve.func1()
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:1854 +0xb0
panic({0x103e95780, 0x105020ba0})
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/runtime/panic.go:890 +0x258
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).DecodeRaw(0x0, {{0x14000c23c00, 0x364, 0x380}, {0x0, 0x0}}, {0x104139af0, 0x14000812b00})
        /Users/krkr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/decode.go:76 +0xec
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Decoder).Decode(_, {{{0x1400063a540, 0x24}, {{0x1400067aa38, 0x13}, {0x140010fe270, 0x7}, {0x140010fe278, 0x4}}, {{0x1400067aa50, ...}, ...}, ...}}, ...)
        /Users/krkr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/decode.go:49 +0x74
github.com/elastic/cloud-on-k8s/v2/pkg/controller/common/webhook.(*validatingWebhook).Handle(_, {_, _}, {{{0x1400063a540, 0x24}, {{0x1400067aa38, 0x13}, {0x140010fe270, 0x7}, {0x140010fe278, ...}}, ...}})
        /Users/krkr/dev/src/github.com/elastic/cloud-on-k8s/pkg/controller/common/webhook/webhook.go:76 +0x23c
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle(_, {_, _}, {{{0x1400063a540, 0x24}, {{0x1400067aa38, 0x13}, {0x140010fe270, 0x7}, {0x140010fe278, ...}}, ...}})
        /Users/krkr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/webhook.go:169 +0x18c
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP(0x14000062dc0, {0x12d0b4088?, 0x14000cad4a0}, 0x14000630600)
        /Users/krkr/dev/pkg/mod/sigs.k8s.io/controller-runtime@v0.15.0/pkg/webhook/admission/http.go:98 +0x9bc
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerInFlight.func1({0x12d0b4088, 0x14000cad4a0}, 0x14000aa5f00?)
        /Users/krkr/dev/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:60 +0xb0
net/http.HandlerFunc.ServeHTTP(0x10414ce50?, {0x12d0b4088?, 0x14000cad4a0?}, 0x14000995818?)
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:2122 +0x38
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1({0x10414ce50?, 0x14000228460?}, 0x14000630600)
        /Users/krkr/dev/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:147 +0xa0
net/http.HandlerFunc.ServeHTTP(0x14000995a58?, {0x10414ce50?, 0x14000228460?}, 0x14000590d40?)
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:2122 +0x38
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerDuration.func2({0x10414ce50, 0x14000228460}, 0x14000630600)
        /Users/krkr/dev/pkg/mod/github.com/prometheus/client_golang@v1.15.1/prometheus/promhttp/instrument_server.go:109 +0x94
net/http.HandlerFunc.ServeHTTP(0x14000995a98?, {0x10414ce50?, 0x14000228460?}, 0x102739e58?)
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:2122 +0x38
net/http.(*ServeMux).ServeHTTP(0x140004f06c0?, {0x10414ce50, 0x14000228460}, 0x14000630600)
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:2500 +0x140
net/http.serverHandler.ServeHTTP({0x10413ebb8?}, {0x10414ce50, 0x14000228460}, 0x14000630600)
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:2936 +0x2d8
net/http.(*conn).serve(0x14000976d80, {0x10414def0, 0x1400034e000})
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:1995 +0x560
created by net/http.(*Server).Serve
        /opt/homebrew/Cellar/go/1.20.3/libexec/src/net/http/server.go:3089 +0x520

[thbkrkr]

How to run the operator locally and be able to test a webhook?

Enable the webhook server:

diff --git a/Makefile b/Makefile
index 45b9dbde7..9c19f92a4 100644
--- a/Makefile
+++ b/Makefile
@@ -211,6 +211,7 @@ go-run:
                        -tags "$(GO_TAGS)" \
                        ./cmd/main.go manager \
                                --development \
+                               --enable-webhook \
                                --enable-leader-election=false \
                                --log-verbosity=$(LOG_VERBOSITY) \
                                --ca-cert-validity=10h --ca-cert-rotate-before=1h \

Start a tunnel and copy the generated public URL:

> ngrok http https://localhost:9443
...
...
Forwarding                    https://abcd-01-23-345-678.xz.ngrok.io -> https://localhost:9443

Change the clientConfig with this URL:

diff --git a/config/webhook/manifests.yaml b/config/webhook/manifests.yaml
index b686e2fae..1a4ee77a1 100644
--- a/config/webhook/manifests.yaml
+++ b/config/webhook/manifests.yaml
@@ -74,10 +74,7 @@ webhooks:
   - v1
   - v1beta1
   clientConfig:
-    service:
-      name: webhook-service
-      namespace: system
-      path: /validate-beat-k8s-elastic-co-v1beta1-beat
+    url: https://abcd-01-23-345-678.xz.ngrok.io/validate-beat-k8s-elastic-co-v1beta1-beat
   failurePolicy: Ignore
   matchPolicy: Exact
   name: elastic-beat-validation-v1beta1.k8s.elastic.co

[barkbay]

TIL about ngrok.io, I used to interact directly with the webhook by forging an AdmissionReview: https://gist.github.com/barkbay/7405c4745d92efe0734110db04e11333

Not sure if it would have help in that case though...