delta perfdata counters

[cbnorman]

I have been writing custom checks for a while now and more recently utilizing performance data. I could be going about this the wrong way so id welcome anyone pointing me in the right direction.

basically i have a number of checks ive written which query things like switch interfaces, the problem is every time the check runs i each metric is since the switch or server last rebooted and i need to write thresholds based on the last time the check completed. so currently i have resorted to querying the icinga API to get the last recorded metrics of that service, running the check and subtracting the icinga API result from the new result creating the delta. i then send the result agian using the icinga2 API as a passive check.

two points id like to improve:

1. calling the API twice in my check.
2. working out the delta in my checks

Ideally id like to specify when sending the new metric that its a counter that id like the delta recorded. using the UOM 'c' doesnt seem to really do anything. if there anything i can do to improve this process?

thanks in advance.

[dnsmichi]

I'm wondering how this relates to Icinga 2. That sounds more of a plugin logic question, and without knowing much details about it, hard to answer. I suggest to ask on the community channels how others deal with that problem.

[tobiasvdk]

Maybe you can add parameters to your plugin which have the values of the last check result performance data. A quick test with the dummy CheckCommand:

apply Service "test-check-result" {
  import "generic-service"

  check_command = "dummy"
  vars.dummy_text = {{ "$service.last_check_result.execution_start$" }}

  assign where host.address
}

The dummy command does not have performance data, so you have to play around a little bit. For the CheckResult type see https://docs.icinga.com/icinga2/latest/doc/module/icinga2/toc#!/icinga2/latest/doc/module/icinga2/chapter/value-types#value-types-checkresult

[cbnorman]

thats sort of what im doing now now, using those macros from the initial get request, except i then minus from the new result before sending. im looking for an icinga function so i could do something like

"performance_data": {metric1 = (new_value1 -$service.last_check_result.performance_data.label == 'metric1').value};0;0;0, metric2 = (new-value2 -$service.last_check_result.performance_data.label == 'metric2').value};0;0;0

[cbnorman]

alternatively I'm wondering if it would be preferable to simply send the perfdata with no service state (if thats possible). icinga2 writes that to influx/grafana and then one the 2 updates the service state when it matches a threshold?

[Stefar77]

I would use Memcached or Redis for this (Only use API if there is no hit from cache). Or just try and push the last state / performance data back to the plugin in a parameter...

[dnsmichi]

This is best to be solved with external fast storage resources. I don't think that there's a reliable solution for a built-in access.