Use prefered naming scheme for prometheus metrics

[AlexanderThaller]

The naming scheme is outlined here https://prometheus.io/docs/practices/naming/

It would be good to do that especially for certain metrics that include the nomad node name.

Example is nomad_nomad0_client_allocs_cpu_throttled_periods.

It should rather export the name as a label like this
nomad_client_allocs_cpu_throttled_periods{node_name="nomad0"}

This would make it much easier to aggregate metrics later.

[chelseakomlo]

Hi, thanks for submitting this issue. What version of Nomad are you running? Have you looked at metrics emitted post 0.7? https://www.nomadproject.io/docs/agent/telemetry.html

[hamann]

@chelseakomlo we see the same with nomad 0.7 and prometheus configuration as shown here

/ # nomad version
Nomad v0.7.0
/ # curl -v https://10.242.0.5:4646/v1/metrics?format=prometheus
....
go_memstats_stack_sys_bytes 2.78528e+06
# HELP go_memstats_sys_bytes Number of bytes obtained from system.
# TYPE go_memstats_sys_bytes gauge
go_memstats_sys_bytes 2.4365304e+07
# HELP go_threads Number of OS threads created.
# TYPE go_threads gauge
go_threads 22
# HELP nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_cpu nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_cpu
# TYPE nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_cpu gauge
nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_cpu{datacenter="gc-europe-west1",node_id="79460472-c092-4386-150b-3dfef8dd4687"} 160
# HELP nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_disk nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_disk
# TYPE nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_disk gauge
nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_disk{datacenter="gc-europe-west1",node_id="79460472-c092-4386-150b-3dfef8dd4687"} 800
# HELP nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_iops nomad_agent_1_staging_c_nextjournal_com_internal_client_allocated_iops
....

Screenshot from prometheus sources:

[AlexanderThaller]

We also use the latest version (0.7)

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.