Expose concurrency based metrics

[dhumphries-sainsburys]

In a separate issue we are looking at issues with concurrency but it is hard to get a true grasp of any problem real or perceived without concrete metrics. We have some for things interacting with flux but the actual inner workings of things such as the kustomize-controller are sadly hidden from us so we are left guessing at what is happening with. Is it possible to expose some useful concurrency metrics to prometheus such as:

queue depth
thread timings
active thread counts

Whilst the above are things that we think could be useful for our current state more metrics wouldn't be a bad thing (even better if we could also turn off any that aren't needed)

[stefanprodan]

We expose all the Go runtime metrics, you can find there metrics about go routines (there are no threads in Go), heap size, etc.

[dhumphries-sainsburys]

Having had a look i can see the go metrics as suggested and turns out there is metrics for threads in go called go_threads from prometheus which aligns with the concurrency option (18 + concurrency value) although it also seems that the count isn't so useful as it just puts them into a wait when doing nothing so you can't see how busy it is from the metric.

The more valuable metrics we need which i don't see exposed anywhere is something like a queue depth as we know we are pushing kustomize with a lot of work when our webhooks fire but we have no idea how far through the queue it is at any time we just have to watch the status field of the objects and as they only update when actively being reconciled the exposed metric of gotk_reconcile_condition isn't all that useful for knowing the current state