add node_status and node_scheduling_eligibility labels to client_alloc_allocated metrics

[gmichalec-pandora]

For reporting security vulnerabilities please refer to the website.

If you have a question, prepend your issue with [question] or preferably use the nomad mailing list.

If filing a bug please include the following:

Nomad version

Output from nomad version
Nomad v0.10.4+ent

Operating system and Environment details

Debian GNU/Linux 9.9 (stretch)

Issue

The addition of node_status and node_scheduling_eligibility (#6130) is a great step, but in order for us to properly monitor cluster usage and scheduling availability, it would be incredibly useful to add these labels to the metrics for client allocated resources. For example, we have a grafana charting the % of available cpu resources in our cluster - however, we currently have no way to account for what hosts are ineligible/down in that chart. If we could get those labels added to the following metrics, it would be a huge help:

• nomad_client_allocated_cpu
• nomad_client_unallocated_cpu
• nomad_client_allocated_memory
• nomad_client_unallocated_memory
• nomad_client_allocated_disk
• nomad_client_unallocated_disk
• nomad_client_allocated_network
• nomad_client_unallocated_network

Thank you!

[notnoop]

Thanks - this is a reasonable request and would be useful indeed to expose these for meaningful charts and alerts.

[gmichalec-pandora]

Just an update here - I've realized I can accomplish this task through some fancy PromQL:
sum(nomad_client_unallocated_cpu and on (host) nomad_client_uptime{node_scheduling_eligibility="eligible"})
but it still would be nice to have the labels directly on those metrics!