add beacon checks metrics

[voyvodov]

This introduce metrics for Salt status beacons
(https://docs.saltproject.io/en/latest/ref/beacons/all/salt.beacons.status.html) Will expose metrics for last beacon event per minion and also the total number of observed minions (minions not reported for 4 hours by default are removed every hour).
This way, user can not only monitor jobs success, but also if all expected minions are reporting back to Salt master.
The exporter will watch the PKI directory of the master for changed files and this way will know how many minions are expected to respond back. On (re)start it will read the minions' keys for initial status of observable minions.

[voyvodov]

If the idea is okey will also update the docs with information how this can be used and what is required from Salt side to be done to work correctly

[kpetremann]

Hi @voyvodov,

Thanks for the PR and the idea. I'll have a look!

[kpetremann]

I like the idea :)
I'll start the review of the code as soon as possible.

minions not reported for 4 hours by default are removed every hour

It should be configurable by the user.

The exporter will watch the PKI directory of the master for changed files and this way will know how many minions are expected to respond back.

Could also be used to expose the number of registered minion.

This way, user can not only monitor jobs success

nitpick on this: there is a metric to get the total of minion responses, whether it is successful or not. But beacon is a better way 👍

[voyvodov]

Ohh... I put wrong description.

In fact it will not remove them until they are removed fron salt master.
Initially it was using beacon to detect minions and background thread to cleanup (from which is the description). After a while I change it to get the public keys in the PKI directory of the accepted minions and a thread watching for changes (e.g. accepted or deleted minions).
So, exporter will just report last "heartbeat". It's up to the user what value will choose to alert in Grafana or Prometheus.

There is a metric which report the total of all registered minions (salt_health_minions_total)

[kpetremann]

@voyvodov LGTM thanks for the contrib!

If the idea is okey will also update the docs with information how this can be used and what is required from Salt side to be done to work correctly

do you want to update this PR with the documentation or do a dedicated PR for this?

[voyvodov]

@voyvodov LGTM thanks for the contrib!

If the idea is okey will also update the docs with information how this can be used and what is required from Salt side to be done to work correctly

do you want to update this PR with the documentation or do a dedicated PR for this?

If you're okey I can do it in seperate PR in next few days

[kpetremann]

ok no problemo :) let's merge it!

I'll release a new version of the exporter when we will have the documentation up to date.