Local queues prometheus metrics

[astefanutti]

What would you like to be added:

Expose Prometheus metrics for local queues, equivalent to the existing cluster queue metrics, but filtered and labeled by local queues.

Similarly to the visibility API, that serves information about pending workloads in local queues, it would be possible to get metrics like like pending workloads, admitted active workloads, resource usage, etc, for local queues.

If cardinality is a concern, those metrics could be exposed behind a feature flag.

Why is this needed:

Metrics about local queues can be useful for the batch users persona, so they can have visibility and historical trends about their workloads.

While some metrics are already available for cluster queues, exposing them to the batch users persona presents the following challenges / limitations:

• Cluster queues metrics are global and cannot be filtered by namespaces / tenants
• Querying "cluster-scoped" metrics in secured Prometheus instances is generally only authorised for cluster admin users that have access to all namespaces / tenants

Completion requirements:

This enhancement requires the following artifacts:

• Design doc
• API change
• Docs update

The artifacts should be linked in subsequent comments.

[astefanutti]

@alculquicondor @tenzen-y Do you think that'd be a useful / possible enhancement?

[alculquicondor]

If cardinality is a concern, those metrics could be exposed behind a feature flag.

Yes, that's the primary concern. I wouldn't make it a feature flag, but a long-term configuration field.

[tenzen-y]

@alculquicondor @tenzen-y Do you think that'd be a useful / possible enhancement?

I understand that this feature is so useful, but I have the same concern with @alculquicondor.
IIRC, previously, we had similar discussions when we designed Visibility/Visibility On-demand.
So, if we introduce this feature, configurable this feature by Config API would be better.

Anyway, I guess that having a small KEP would be better since we may extend the existing Config API.

[astefanutti]

That makes sense. There could be one or two options added to the Config API, similar to the existing .metrics.enableClusterQueueResources, like .metrics.enableLocalQueues and .metrics.enableLocalQueueResources.

I can work on a small KEP if you guys give the green light.

[alculquicondor]

It seems simple enough.

[tenzen-y]

SGTM

[astefanutti]

Thanks for your quick feedback! I'll work on it asap.

/assign

[k8s-triage-robot]

The Kubernetes project currently lacks enough contributors to adequately respond to all issues.

This bot triages un-triaged issues according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue as fresh with /remove-lifecycle stale
• Close this issue with /close
• Offer to help out with Issue Triage

Please send feedback to sig-contributor-experience at kubernetes/community.

/lifecycle stale

[alculquicondor]

@astefanutti are you still looking into this?

[astefanutti]

@alculquicondor I haven't, but hopefully we'll get back to it soon.

[astefanutti]

/remove-lifecycle stale

[astefanutti]

/unassign

[alculquicondor]

@varshaprasad96, please write /assign in a comment to claim this issue. It's important to communicate that you are working on an issue so that other contributors don't try to work on the same thing.

[varshaprasad96]

/assign