workqueue metrics are unavailable likely due to an upstream change to the controller-runtime

[sy-be]

Component(s)

operator

What happened?

Description

Operator was exposing controller runtime metrics such as workqueue_depth which is also used in the OTEL operator helm chart to setup prometheus alerts, but it looks like this metric isn't available in recent opentelemetry-operator versions.

Metrics are defined in this package.

I've found some upstream issues/PRs which may or may not be related to the issue:

• Don't set global workqueue MetricsProvider kubernetes-sigs/controller-runtime#2238
• Create a separate registry for work-queue metrics kubernetes-sigs/controller-runtime#2670
• Allow non-global metrics factory for workqueues kubernetes/client-go#1193
• ✨ Add a separate registry for workqueue metrics kubernetes-sigs/controller-runtime#2671

Steps to Reproduce

• Install opentelemetry-operator in a k8s cluster
• Make a GET request to its service, e.g. `curl http://<service_name_or_ip_address>:8080/metrics

Expected Result

Various workqueue prefixed metrics, such as workqueue_depth, workqueue_adds_total, workqueue_retries_total, etc.

Actual Result

Observe that there are no workqueue prefixed metrics

Kubernetes Version

1.24.15

Operator version

v0.92.1

Collector version

v0.92.0

Environment information

No response

Log output

curl http://172.30.35.141:8080/metrics 
# HELP certwatcher_read_certificate_errors_total Total number of certificate read errors
# TYPE certwatcher_read_certificate_errors_total counter
certwatcher_read_certificate_errors_total 0
# HELP certwatcher_read_certificate_total Total number of certificate reads
# TYPE certwatcher_read_certificate_total counter
certwatcher_read_certificate_total 1
# HELP controller_runtime_webhook_latency_seconds Histogram of the latency of processing admission requests
# TYPE controller_runtime_webhook_latency_seconds histogram
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.005"} 2263
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.01"} 2270
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.025"} 2272
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.05"} 2272
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.1"} 2272
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.25"} 2272
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="0.5"} 2273
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="1"} 2273
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="2.5"} 2273
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="5"} 2273
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="10"} 2273
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation",le="+Inf"} 2273
controller_runtime_webhook_latency_seconds_sum{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation"} 2.9031210569999986
controller_runtime_webhook_latency_seconds_count{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation"} 2273
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.005"} 2236
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.01"} 2237
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.025"} 2237
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.05"} 2237
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.1"} 2237
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.25"} 2237
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.5"} 2239
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="1"} 2239
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="2.5"} 2239
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="5"} 2239
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="10"} 2239
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="+Inf"} 2239
controller_runtime_webhook_latency_seconds_sum{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 2.7003286010000007
controller_runtime_webhook_latency_seconds_count{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 2239
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.005"} 57331
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.01"} 57589
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.025"} 57692
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.05"} 57708
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.1"} 57712
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.25"} 57716
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="0.5"} 57724
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="1"} 57730
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="2.5"} 57731
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="5"} 57731
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="10"} 57731
controller_runtime_webhook_latency_seconds_bucket{webhook="/mutate-v1-pod",le="+Inf"} 57731
controller_runtime_webhook_latency_seconds_sum{webhook="/mutate-v1-pod"} 79.00923619699945
controller_runtime_webhook_latency_seconds_count{webhook="/mutate-v1-pod"} 57731
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.005"} 2270
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.01"} 2275
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.025"} 2275
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.05"} 2275
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.1"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.25"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="0.5"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="1"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="2.5"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="5"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="10"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation",le="+Inf"} 2276
controller_runtime_webhook_latency_seconds_sum{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation"} 1.9971447300000038
controller_runtime_webhook_latency_seconds_count{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation"} 2276
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.005"} 2364
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.01"} 2364
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.025"} 2364
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.05"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.1"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.25"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="0.5"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="1"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="2.5"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="5"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="10"} 2367
controller_runtime_webhook_latency_seconds_bucket{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector",le="+Inf"} 2367
controller_runtime_webhook_latency_seconds_sum{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 1.8726208400000002
controller_runtime_webhook_latency_seconds_count{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 2367
# HELP controller_runtime_webhook_requests_in_flight Current number of admission requests being served.
# TYPE controller_runtime_webhook_requests_in_flight gauge
controller_runtime_webhook_requests_in_flight{webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation"} 0
controller_runtime_webhook_requests_in_flight{webhook="/mutate-opentelemetry-io-v1alpha1-opampbridge"} 0
controller_runtime_webhook_requests_in_flight{webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 0
controller_runtime_webhook_requests_in_flight{webhook="/mutate-v1-pod"} 0
controller_runtime_webhook_requests_in_flight{webhook="/validate-opentelemetry-io-v1alpha1-instrumentation"} 0
controller_runtime_webhook_requests_in_flight{webhook="/validate-opentelemetry-io-v1alpha1-opampbridge"} 0
controller_runtime_webhook_requests_in_flight{webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 0
# HELP controller_runtime_webhook_requests_total Total number of admission requests by HTTP status code.
# TYPE controller_runtime_webhook_requests_total counter
controller_runtime_webhook_requests_total{code="200",webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation"} 2273
controller_runtime_webhook_requests_total{code="200",webhook="/mutate-opentelemetry-io-v1alpha1-opampbridge"} 0
controller_runtime_webhook_requests_total{code="200",webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 2239
controller_runtime_webhook_requests_total{code="200",webhook="/mutate-v1-pod"} 57731
controller_runtime_webhook_requests_total{code="200",webhook="/validate-opentelemetry-io-v1alpha1-instrumentation"} 2276
controller_runtime_webhook_requests_total{code="200",webhook="/validate-opentelemetry-io-v1alpha1-opampbridge"} 0
controller_runtime_webhook_requests_total{code="200",webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 2367
controller_runtime_webhook_requests_total{code="500",webhook="/mutate-opentelemetry-io-v1alpha1-instrumentation"} 0
controller_runtime_webhook_requests_total{code="500",webhook="/mutate-opentelemetry-io-v1alpha1-opampbridge"} 0
controller_runtime_webhook_requests_total{code="500",webhook="/mutate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 0
controller_runtime_webhook_requests_total{code="500",webhook="/mutate-v1-pod"} 0
controller_runtime_webhook_requests_total{code="500",webhook="/validate-opentelemetry-io-v1alpha1-instrumentation"} 0
controller_runtime_webhook_requests_total{code="500",webhook="/validate-opentelemetry-io-v1alpha1-opampbridge"} 0
controller_runtime_webhook_requests_total{code="500",webhook="/validate-opentelemetry-io-v1alpha1-opentelemetrycollector"} 0
# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
# TYPE go_gc_duration_seconds summary
go_gc_duration_seconds{quantile="0"} 8.7435e-05
go_gc_duration_seconds{quantile="0.25"} 0.000115646
go_gc_duration_seconds{quantile="0.5"} 0.000136136
go_gc_duration_seconds{quantile="0.75"} 0.000161084
go_gc_duration_seconds{quantile="1"} 0.006963877
go_gc_duration_seconds_sum 2.709871808
go_gc_duration_seconds_count 10828
# HELP go_goroutines Number of goroutines that currently exist.
# TYPE go_goroutines gauge
go_goroutines 53
# HELP go_info Information about the Go environment.
# TYPE go_info gauge
go_info{version="go1.21.5"} 1
# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
# TYPE go_memstats_alloc_bytes gauge
go_memstats_alloc_bytes 2.78195136e+08
# HELP go_memstats_alloc_bytes_total Total number of bytes allocated, even if freed.
# TYPE go_memstats_alloc_bytes_total counter
go_memstats_alloc_bytes_total 1.17050730904e+11
# HELP go_memstats_buck_hash_sys_bytes Number of bytes used by the profiling bucket hash table.
# TYPE go_memstats_buck_hash_sys_bytes gauge
go_memstats_buck_hash_sys_bytes 2.500882e+06
# HELP go_memstats_frees_total Total number of frees.
# TYPE go_memstats_frees_total counter
go_memstats_frees_total 4.90944089e+08
# HELP go_memstats_gc_sys_bytes Number of bytes used for garbage collection system metadata.
# TYPE go_memstats_gc_sys_bytes gauge
go_memstats_gc_sys_bytes 1.6520592e+07
# HELP go_memstats_heap_alloc_bytes Number of heap bytes allocated and still in use.
# TYPE go_memstats_heap_alloc_bytes gauge
go_memstats_heap_alloc_bytes 2.78195136e+08
# HELP go_memstats_heap_idle_bytes Number of heap bytes waiting to be used.
# TYPE go_memstats_heap_idle_bytes gauge
go_memstats_heap_idle_bytes 1.62422784e+08
# HELP go_memstats_heap_inuse_bytes Number of heap bytes that are in use.
# TYPE go_memstats_heap_inuse_bytes gauge
go_memstats_heap_inuse_bytes 3.18513152e+08
# HELP go_memstats_heap_objects Number of allocated objects.
# TYPE go_memstats_heap_objects gauge
go_memstats_heap_objects 2.623269e+06
# HELP go_memstats_heap_released_bytes Number of heap bytes released to OS.
# TYPE go_memstats_heap_released_bytes gauge
go_memstats_heap_released_bytes 1.36101888e+08
# HELP go_memstats_heap_sys_bytes Number of heap bytes obtained from system.
# TYPE go_memstats_heap_sys_bytes gauge
go_memstats_heap_sys_bytes 4.80935936e+08
# HELP go_memstats_last_gc_time_seconds Number of seconds since 1970 of last garbage collection.
# TYPE go_memstats_last_gc_time_seconds gauge
go_memstats_last_gc_time_seconds 1.7090529599508982e+09
# HELP go_memstats_lookups_total Total number of pointer lookups.
# TYPE go_memstats_lookups_total counter
go_memstats_lookups_total 0
# HELP go_memstats_mallocs_total Total number of mallocs.
# TYPE go_memstats_mallocs_total counter
go_memstats_mallocs_total 4.93567358e+08
# HELP go_memstats_mcache_inuse_bytes Number of bytes in use by mcache structures.
# TYPE go_memstats_mcache_inuse_bytes gauge
go_memstats_mcache_inuse_bytes 153600
# HELP go_memstats_mcache_sys_bytes Number of bytes used for mcache structures obtained from system.
# TYPE go_memstats_mcache_sys_bytes gauge
go_memstats_mcache_sys_bytes 156000
# HELP go_memstats_mspan_inuse_bytes Number of bytes in use by mspan structures.
# TYPE go_memstats_mspan_inuse_bytes gauge
go_memstats_mspan_inuse_bytes 5.846904e+06
# HELP go_memstats_mspan_sys_bytes Number of bytes used for mspan structures obtained from system.
# TYPE go_memstats_mspan_sys_bytes gauge
go_memstats_mspan_sys_bytes 6.094704e+06
# HELP go_memstats_next_gc_bytes Number of heap bytes when next garbage collection will take place.
# TYPE go_memstats_next_gc_bytes gauge
go_memstats_next_gc_bytes 5.38773376e+08
# HELP go_memstats_other_sys_bytes Number of bytes used for other system allocations.
# TYPE go_memstats_other_sys_bytes gauge
go_memstats_other_sys_bytes 1.1101718e+07
# HELP go_memstats_stack_inuse_bytes Number of bytes in use by the stack allocator.
# TYPE go_memstats_stack_inuse_bytes gauge
go_memstats_stack_inuse_bytes 5.603328e+06
# HELP go_memstats_stack_sys_bytes Number of bytes obtained from system for stack allocator.
# TYPE go_memstats_stack_sys_bytes gauge
go_memstats_stack_sys_bytes 5.603328e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 5.2291316e+08
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 71
# HELP leader_election_master_status Gauge of if the reporting system is master of the relevant lease, 0 indicates backup, 1 indicates master. 'name' is the string used to identify the lease. Please make sure to group by name.
# TYPE leader_election_master_status gauge
leader_election_master_status{name="9f7554c3.opentelemetry.io"} 0
# HELP process_cpu_seconds_total Total user and system CPU time spent in seconds.
# TYPE process_cpu_seconds_total counter
process_cpu_seconds_total 4561.55
# HELP process_max_fds Maximum number of open file descriptors.
# TYPE process_max_fds gauge
process_max_fds 1.048576e+06
# HELP process_open_fds Number of open file descriptors.
# TYPE process_open_fds gauge
process_open_fds 17
# HELP process_resident_memory_bytes Resident memory size in bytes.
# TYPE process_resident_memory_bytes gauge
process_resident_memory_bytes 4.171776e+08
# HELP process_start_time_seconds Start time of the process since unix epoch in seconds.
# TYPE process_start_time_seconds gauge
process_start_time_seconds 1.70774102217e+09
# HELP process_virtual_memory_bytes Virtual memory size in bytes.
# TYPE process_virtual_memory_bytes gauge
process_virtual_memory_bytes 1.907625984e+09
# HELP process_virtual_memory_max_bytes Maximum amount of virtual memory available in bytes.
# TYPE process_virtual_memory_max_bytes gauge
process_virtual_memory_max_bytes 1.8446744073709552e+19

### Additional context

_No response_