adds WatchList Latency to APIResponsivenessPrometheus #2764
Conversation
k8s-ci-robot
added
the
kind/feature
k8s-ci-robot
added
cncf-cla: yes
| watchListLatencyMetricName = "apiserver_watch_list_duration_seconds" | ||
| // watchListLatencyQuery placeholders must be replaced with (1) quantile (2) query window size | ||
| watchListLatencyQuery = "histogram_quantile(%.2f, sum(rate(%v_bucket{}[%v])) by (group, version, resource, scope, le))" | ||
|
|
There was a problem hiding this comment.
Looking into test results:
https://storage.googleapis.com/kubernetes-jenkins/pr-logs/pull/perf-tests/2764/pull-perf-tests-clusterloader2/1810319004401668096/artifacts/APIResponsivenessPrometheus_simple_load_2024-07-08T14:50:01Z.json
I see only one entry for watchlist (pod list on namespace scope).
Is that expected? What is issueing this request?
There was a problem hiding this comment.
good question.
do we have metrics so that i could execute the prom query manually ?
could it be that the run was using a server with enabled watchlist feature ?
There was a problem hiding this comment.
could it be that the run was using a server with enabled watchlist feature ?
it looks like this was our case, the most recent run doesn't have any entries of watchlist (the feature was turned off on the server some time ago)
There was a problem hiding this comment.
So I was wondering why we've seen only one such entry, but I guess it's because it's disabled in client-go by default (and was enabled only in KCM). So that makes sense.
| } | ||
|
|
||
| func (ex *fakeQueryExecutor) Query(query string, _ time.Time) ([]*model.Sample, error) { | ||
|
|
clusterloader2/pkg/measurement/common/slos/api_responsiveness_prometheus_test.go
Show resolved
Hide resolved
|
/test perf-tests-clusterloader2 |
|
@p0lyn0mial: The specified target(s) for
The following commands are available to trigger optional jobs:
Use
In response to this:
Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. |
|
/test pull-perf-tests-clusterloader2 |
p0lyn0mial
force-pushed
the
upstream-api-responsiveness-watch-list
branch
from
71544a3 to
bead663
Compare
|
/lgtm |
k8s-ci-robot
added
the
lgtm
|
[APPROVALNOTIFIER] This PR is APPROVED This pull-request has been approved by: p0lyn0mial, wojtek-t The full list of commands accepted by this bot can be found here. The pull request process is described here
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
k8s-ci-robot
added
the
approved
What type of PR is this?
/kind feature
What this PR does / why we need it:
WatchListlatency is gathered for 50th, 90th and 99th duration quantiles for watch list requests broken down by group, resource, scope.The new metric (kubernetes/kubernetes#120490) allows for comparing watch-list requests with standard list requests and measuring performance of the new requests in general.
Which issue(s) this PR fixes:
Fixes #
Special notes for your reviewer: