[BUG] addPrometheusConfig in the mutating webhook always panics

[Tom-Newton]

Description

Every time the pod webhook runs it causes a panic in the mutating webhook logs.

• ✋ I have searched the open/closed issues and my issue is not listed.

Reproduction Code [Required]

I'm pretty sure running any SparkApplication will reproduce.

Steps to reproduce the behavior:
Create a SparkApplication then look at the webhook logs

Expected behavior

All the expected mutations happen and no errors are logged.

Actual behavior

Logs

2024-10-03T18:11:01.950Z	INFO	webhook/sparkpod_defaulter.go:339	could not mount volume %s in path %s	{"delegated-gen2-enrichment-targetless-vehic-863635-prom-conf-vol": "/etc/metrics/conf"}
2024-10-03T18:11:01.950Z	DPANIC	webhook/sparkpod_defaulter.go:341	odd number of arguments passed as key-value pairs for logging	{"ignored key": 1099}
github.com/kubeflow/spark-operator/internal/webhook.addPrometheusConfig
	/workspace/internal/webhook/sparkpod_defaulter.go:341
github.com/kubeflow/spark-operator/internal/webhook.mutateSparkPod
	/workspace/internal/webhook/sparkpod_defaulter.go:136
github.com/kubeflow/spark-operator/internal/webhook.(*SparkPodDefaulter).Default
	/workspace/internal/webhook/sparkpod_defaulter.go:92
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*defaulterForType).Handle
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.5/pkg/webhook/admission/defaulter_custom.go:80
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).Handle
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.5/pkg/webhook/admission/webhook.go:169
sigs.k8s.io/controller-runtime/pkg/webhook/admission.(*Webhook).ServeHTTP
	/go/pkg/mod/sigs.k8s.io/controller-runtime@v0.17.5/pkg/webhook/admission/http.go:119
sigs.k8s.io/controller-runtime/pkg/webhook/internal/metrics.InstrumentedHook.InstrumentHandlerInFlight.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:60
net/http.HandlerFunc.ServeHTTP
	/usr/local/go/src/net/http/server.go:2220
github.com/prometheus/client_golang/prometheus/promhttp.InstrumentHandlerCounter.func1
	/go/pkg/mod/github.com/prometheus/client_golang@v1.19.1/prometheus/promhttp/instrument_server.go:147
net/http.HandlerFunc.ServeHTTP

I think the important part is webhook/sparkpod_defaulter.go:341 odd number of arguments passed as key-value pairs for logging {"ignored key": 1099}.

In addition to logging an error I suspect these mutations are not happening. Finally, I'm pretty sure its logging "could not" after succeeding, so that is a bit confusing.

Environment & Versions

• Spark Operator App version: 2.0.1
• Helm Chart Version: 2.0.1
• Kubernetes Version: 1.29.7
• Apache Spark version: 3.5.0

Additional context