Documentation/op-guide: fix failed RPC rate, leader election metrics #8093
Conversation
Documentation/op-guide/grafana.json
Outdated
| @@ -922,7 +922,7 @@ | |||
| "stack": false, | |||
| "steppedLine": false, | |||
| "targets": [{ | |||
| "expr": "etcd_server_leader_changes_seen_total", | |||
| "expr": "delta(etcd_server_leader_changes_seen_total[1m])", | |||
There was a problem hiding this comment.
this is wrong. we still want to see the total.
|
@brancz can you do a double check on the rules? thanks! |
gyuho
force-pushed
the
grafana
branch
3 times, most recently
from
943a113
to
01644b1
Compare
|
@xiang90 Removed election query change, instead changed the title. |
Documentation/op-guide/grafana.json
Outdated
| @@ -924,15 +924,15 @@ | |||
| "targets": [{ | |||
| "expr": "etcd_server_leader_changes_seen_total", | |||
| "intervalFactor": 2, | |||
| "legendFormat": "{{instance}} Leader Change Seen", | |||
| "legendFormat": "{{instance}} Total Leader Elections", | |||
There was a problem hiding this comment.
I think we can actually change this to Leader Change Seen in one day. and change [1m] to [1day]
There was a problem hiding this comment.
Just switched to 1 day with changes function.
gyuho
force-pushed
the
grafana
branch
2 times, most recently
from
3618884
to
02e7257
Compare
Codecov Report
@@ Coverage Diff @@
## master #8093 +/- ##
=========================================
Coverage ? 76.71%
=========================================
Files ? 342
Lines ? 26568
Branches ? 0
=========================================
Hits ? 20381
Misses ? 4739
Partials ? 1448Continue to review full report at Codecov.
|
Documentation/op-guide/grafana.json
Outdated
| @@ -115,17 +115,17 @@ | |||
| "stack": false, | |||
| "steppedLine": false, | |||
| "targets": [{ | |||
| "expr": "sum(rate(grpc_server_started_total{grpc_type=\"unary\"} [1m]))", | |||
| "expr": "sum(rate(grpc_server_started_total{grpc_type=\"unary\"}[1m])) by (grpc_method)", | |||
There was a problem hiding this comment.
A typical scrape interval is 15s, 30s or even 60s, in such a case this range query would only rate over 4, 2 or 1 sample. I'd suggest to make this rather 5m.
|
@brancz thanks for the review. |
|
@xiang90 always happy to help 🙂 |
|
lgtm |
Documentation/op-guide/grafana.json
Outdated
| "intervalFactor": 2, | ||
| "legendFormat": "{{instance}} RPC Rate", | ||
| "legendFormat": "{{grpc_method}} RPC Rate", |
There was a problem hiding this comment.
probably also change this to {{instance}}{{grpc_method}} RPC Rate
There was a problem hiding this comment.
we can change the view to a stack view to aggregate them
|
@xiang90 PTAL. |
|
@gyuho Hmm... After second thought, I prefer an aggregated view for both RPC and failed RPC. As discussed with @heyitsanthony before, the dashboard should be as clear and easy as possible. If users want to see detailed information, they can create a perf/debugging page themselves. |
|
All fixed. |
This fixes failed RPC rate query, where we do not need subtraction because we already query by the status code. Also adds grpc_method to make it more specific. Most of the time, the failure recovers within 10-second, which is our Prometheus scrap interval, so 'rate' query might not cover that time window, showing as 0s, but still shows up in the graph. Signed-off-by: Gyu-Ho Lee <gyuhox@gmail.com>
|
lgtm |
This fixes failed RPC rate query, where we do not need
subtraction because we already query by the status code.
Also adds grpc_method to make it more specific. Most of the
time, the failure recovers within 10-second, which is our
Prometheus scrap interval, so 'rate' query might not cover
that time window, showing as 0s, but still shows up in the graph.
Before
After
Also fixes RPC rate query.