-
Notifications
You must be signed in to change notification settings - Fork 1.1k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
[suggestion] Omitting metrics if not success #579
Comments
That a probe failed doesn't mean that the HTTP request failed, the status code could have been wrong for example. |
@brian-brazil that's right, but imagine the following query:
My core point here is - if a particular metric was not actually scraped (for whatever reason: be it a transport error, or http status code error, or any other error) - the complete metric must be removed from the response, not the zero value returned as it's now. |
The TCP connection time doesn't become invalid because a subsequent HTTP request failed, this isn't sane in general. |
That's NOT what I said. I probably am terrible at explaining, but I will try once again. If a particular metric was not actually scraped for whatever reason - it must not be exported, at all, since Returning to my initial example:
Here are the metrics exported for a target with not-existing DNS address. Do you think those zeroes make any sense? |
@brian-brazil I think the point is that exposing TCP connection duration isn't valid if we fail early, for example at URL parsing or DNS lookup. Currently we init all the label values far too early in the setup of the probe. This results in invalid (0) results for data when we haven't even reached the connect phase of the probe. |
Don't init http probe phase labels for phases we haven't observed. Avoids exposing 0 value samples for things we haven't measured. Fixes: #579 Signed-off-by: Ben Kochie <superq@gmail.com>
Don't init http probe phase labels for phases we haven't observed. Avoids exposing 0 value samples for things we haven't measured. Fixes: #579 Signed-off-by: Ben Kochie <superq@gmail.com>
Don't init http probe phase labels until we're ready to send the request. Avoids exposing 0 value samples for things we haven't started to measure. Fixes: #579 Signed-off-by: SuperQ <superq@gmail.com>
Hi, faced the same confusing situation: The metrics return the value 0, but phases, in principle, did not start processing at all (except dns lookup). This set us on the wrong track of latency diagnostics followed by metrics dashboards: It would be much clearer if the phases that did not actually occur for the query were not present in the results but I'm not sure how this compares to the best practices for writing exporters.
|
Don't init http probe phase labels until we're ready to send the request. Avoids exposing 0 value samples for things we haven't started to measure. Fix http resolve timing to report lookup time even if it's an error (ie not found). Fixes: #579 Signed-off-by: SuperQ <superq@gmail.com>
Don't init http probe phase labels until we're ready to send the request. Avoids exposing 0 value samples for things we haven't started to measure. Fix http resolve timing to report lookup time even if it's an error (ie not found). Fixes: #579 Signed-off-by: SuperQ <superq@gmail.com>
How can i get metric probe_http_duration_seconds "time now" and " 24h previous"? |
At the moment if the http probe fails
probe_success=0
- the exporter still exportsWhich is a lie - none of those phases completed in a blazing fast 0 seconds.
Why this is a problem: if you perform any analytics over those metrics (eg percentiles, or min) it would be skewed.
Suggestion: what if the exporter does not export metrics it could not really scrape instead of exporting zero values?
The text was updated successfully, but these errors were encountered: