Produce metrics for consumers

[jcpunk]

What would you like to be added:
Some telemetry information about the daemons running. I'm not sure what metrics would make sense for this daemonset - perhaps the flag stats on the node? But having something I can monitor is handy.

Why is this needed:
I generally don't like to deploy things I can't monitor. Having native telemetry available may also help troubleshoot oddities in the wild.

[marquiz]

Thanks @jcpunk for the suggestion. I think this would definitely be a worthy enhancement.

We would just need to understand what kind of metrics to advertise 🧐 On the nfd-master side we could have some counters and/or gauges identifying e.g. the number of nodes updated, CRDs processed etc. We could also expose the time spent on updating nodes on the nfd-master side and the time spent in doing the feature discovery on the nfd-worker side.

ping @ArangoGutierrez @PiotrProkop @zvonkok @Tal-or any thoughts?

[ArangoGutierrez]

On worker side, maybe advertise if the worker can't reach master, even tho this is logged already.

[ArangoGutierrez]

/assign

[marquiz]

/milestone v0.14

[marquiz]

@ArangoGutierrez is working on an implementation.

@jcpunk @PiotrProkop @zvonkok @fmuyassarov @Tal-or any wishes/thoughts on which kind of metrics to expose. I listed some above

[ArangoGutierrez]

Getting there

[jcpunk]

I'm not really sure what metrics would be useful...

[Tal-or]

@marquiz on the topology-updater side we can expose the following:

1. Number of updates done to NRT objects, each update increase the counter by one.
2. Number of PodresourceAPI calls, each call increases the counter by one.
3. We can also expose the time the update took for each node

Adding @swatisehgal and @ffromani as they might have more ideas on the subject