Metrics

[kzangeli]

Prometheus Style Metrics

A very basic implementation is already available, using the external library prometheus-c-client.
However, the library is no good.
The idea is to replace the library for an in-house implementation.

The library is not "ideal" for a number of reasons:

• Forces MHD to be a shared library (the broker already has it as a static library

• The source code is no good - lots of calls to malloc and other stuff I don't want the broker to use

• It's a "black box" and the broker doesn't know when metrics request comes in - all taken care of by the library.
  May sound like a "good thing" but, it's really not. It means the response needs to be 100% prepared always.
  If instead the broker is aware of when metrics request enters, the request can be prepared only when asked for.
  A counter, instead of being an allocated string, updated thousands of times per second, will just be an incrementing integer.
  The string is only prepared when a metrics request enters. Huge performance gains - metrics could be always ON.
  Also, windows in time, like last 5 min or last 100 of something could be implemented without huge performance losses.
  With the library, I would have to push stuff, keep their timestamps and then remove them. That's just too much ...
  All problems are solved by implementing the metrics inside the broker, and not use the prometheus-c-client library.

• Only if started with -metrics (not needed if in-house - it can be always on, as minor affect on performance)

Counters/Gauges

• Total number of Requests served
• Erroneous incoming requests
• Counters per service? Also per tenant?
• Total number of Notifications sent
• Total number of failed notifications
• Total number of Database Errors
• Start Time
• Uptime
• CPU Load ?
• Memory consumption ?
• Array of counters for responses with status code:
  • 200
  • 201
  • 204
  • 207
  • 2xx (sum of all other 200s)
  • 400
  • 404
  • 4xx (sum of all other 400s)
  • 500
  • 501
  • 5xx