Provide telemetry about allocated / available resources

[stew]

Nomad version

0.5.2

I'm wanting to collect telemetry about how much of my cluster resources have already been allocated, and I don't see any current telemetry which allows me to figure out stuff like "how much memory does my cluster have unallocated" "how much cpu do I have unallocated" "what is the most amount of unallocated memory on any client currently", which is telemetry I feel like I need in order to know when I should be scaling my cluster up.

[OferE]

If i understand correctly, you can calculate it for CPU, Memory and disk space easily by using the following two:
https://www.nomadproject.io/docs/http/client-stats.html
https://www.nomadproject.io/docs/http/client-allocation-stats.html

[dadgar]

@stew Are you asking for each client to report its allocated and available resources? You could then aggregate for a cluster wide view

[stew]

@dadgar that would be ideal, yeah. Does it make sense?

[stew]

@OferE yeah, and you see exactly the stats I'm looking for when you look at the output of nomad node-status $nodeid. I just want them to go out through the statsd telemetry which is already wired into our monitoring systems.

[dadgar]

@stew Yep I was just clarifying to make sure we add the right thing

[dadgar]

@stew Do these stats not work for you: https://github.com/hashicorp/nomad/blob/master/client/client.go#L2147. They don't take into consideration reserved however

[multani]

Adding reserved stats would be cool yes!

[stew]

@dadgar yeah, I do have multiple ways of getting at that stat that work, I guess the reserved stuff is what I'm really after. I want to build stats to help me figure out when I'm close to expecting allocations to get blocked due to a lack of resources.

[burdandrei]

This is what I'm looking for to prevent cluster reservation saturation, maybe AutoScale cluster when less then threshold resources left

[burdandrei]

here is more telemetry per task #2330

[github-actions]

I'm going to lock this issue because it has been closed for 120 days ⏳. This helps our maintainers find and focus on the active issues.
If you have found a problem that seems similar to this, please open a new issue and complete the issue template so we can capture all the details necessary to investigate further.