feat: add salt_state_health gaugeVec

[arsiesys]

Add a new metric salt_state_health that will return 0 or 1 depending of the last time a state was run on the node.
This will allow to easily create alerting rules based on this metric.

For example, in a classic monitoring system, if the last state.highstate failed, it will generate an alert. Once it is back to success, it's going to automatically stop the alarm.

[kpetremann]

hi @arsiesys,

Thanks for the PR.

I had similar metric in the past, but I had to remove it because metrics cardinality was exploding.
I was thinking about adding a similar feature, but with an option to filter on the state/function to watch.

If you are willing to adapt your PR, I would suggest to add a filter in the settings.
Something like: watchedFunctionsPerMinion, watchedStatePerMinion. (there might be a better name)
Values would be a slice of string.

What do you think?

[arsiesys]

I updated my branch to add some configurations options.

The best would be to make it possible to configure a set of association of "functions" with "states". I just did a basic filter for both of them. So this could be improved.

[kpetremann]

Thanks for the PR @arsiesys !!! 🎉 🚀

[kpetremann]

released in v0.1.5

[kpetremann]

I missed one detail, the name of the metric is too precise and does not cover everything.
The metric concerns states, but also functions. I'll update the metric name from salt_state_health to salt_function_status in the next release. (as there is always a function)

note: the prefix status must be used for boolean metrics according Prometheus best practices