-
Notifications
You must be signed in to change notification settings - Fork 1.2k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Changes interval on grafana dashboards to match scrape interval #1669
Changes interval on grafana dashboards to match scrape interval #1669
Conversation
hey @joshleecreates, I was still not able to see the RED metrics in Grafana. |
It worked once for me but I am now seeing the same. I suspect that when it worked I was working off of the branch with the span filtering rules to reduce cardinality, I'll test again after that is merged. |
I'm still seeing issues with Prometheus with the cardinality fix merged:
There's a 23 minute gap before the error appears. I checked in the beginning of that time frame and didn't see any span metrics (or Grafana errors). I think the changes in this PR are necessary for Grafana but there is still something else going on with Prometheus. |
That second issue is different from Prometheus, and it existed before. #1622 is the related issue for it. I pushed a couple of fixes to your branch for that Prometheus issue, and to use a 2m interval instead of 1m. I guess with a 1m metric, we need at least 2 samples for rate to work. |
I think that Prometheus change fixed out of order samples, but now we are getting out of order exemplars.
Some searching tells me this particular thing is not yet solved in Prometheus, and there is an open issue for it. |
I've had this branch running for 4+ days, and Prometheus is stable, with the dashboards working as intended. We still have an error in Prometheus for out of order exemplars, but that is a known issue with Prometheus, that we should track separately. @julianocosta89 can you take another look to see if this works for you? |
🥳 thanks @joshleecreates and @puckpuck! |
Thanks @puckpuck for wrapping this up! |
…-telemetry#1669) * Changes interval on grafana dashboards to match scrape interval * fix out of order sample * use 2m interval for spanmetrics * use 30m for out of order samples --------- Co-authored-by: Juliano Costa <julianocosta89@outlook.com> Co-authored-by: Pierre Tessier <pierre@pierretessier.com>
Changes
Changes interval on the Grafana dashboard to match scrape interval, which fixes the broken visualizations for RED metrics.
(The changes to the axisBorderShow property appear to be from the updated version of Grafana)