Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

feature: add promtheus metrics for yurthub #238

Merged
merged 1 commit into from
Mar 19, 2021
Merged

feature: add promtheus metrics for yurthub #238

merged 1 commit into from
Mar 19, 2021

Conversation

rambohe-ch
Copy link
Member

@rambohe-ch rambohe-ch commented Mar 18, 2021 •
edited
Loading

Ⅰ. Describe what this PR does

add prometheus metrics for yurthub as following:

  • server_healthy_status: healthy status of remote server. 1: healthy, 0: unhealthy
  • in_flight_requests: how many in flight requests are handled by hub agent
  • in_flight_request_counter: counter of in flight requests handled by hub agent
  • closable_conn_counter: counter of underlay tcp connection for hub agent to remote server
  • rejected_request_counter: counter of rejected requests for exceeding in flight limit in hub agent

Ⅱ. Does this pull request fix one issue?

Ⅲ. List the added test cases (unit test/integration test) if any, please explain if no tests are needed.

Ⅳ. Describe how to verify it

  • make release WHAT=cmd/yurthub ARCH=amd64 to build yurthub image
  • run new yurthub image
  • on the edge node, curl http://127.0.0.1:10267/metrics to verify metrics for yurthub, and the metrics output as following:
# HELP node_yurthub_closable_conns_collector collector of underlay tcp connection from hub agent to remote server
# TYPE node_yurthub_closable_conns_collector gauge
node_yurthub_closable_conns_collector{server="xx.xx.xx.xx:6443"} 2
# HELP node_yurthub_in_flight_requests_collector collector of in flight requests handling by hub agent
# TYPE node_yurthub_in_flight_requests_collector gauge
node_yurthub_in_flight_requests_collector{client="kube-proxy",resource="endpoints",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kube-proxy",resource="nodes",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kube-proxy",resource="services",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kubelet",resource="configmaps",subresources="",verb="watch"} 5
node_yurthub_in_flight_requests_collector{client="kubelet",resource="csidrivers",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kubelet",resource="events",subresources="",verb="create"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="events",subresources="",verb="patch"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="leases",subresources="",verb="get"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="leases",subresources="",verb="update"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="nodes",subresources="",verb="get"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="nodes",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kubelet",resource="nodes",subresources="status",verb="patch"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="pods",subresources="",verb="create"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="pods",subresources="",verb="delete"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="pods",subresources="",verb="get"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="pods",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kubelet",resource="pods",subresources="status",verb="patch"} 0
node_yurthub_in_flight_requests_collector{client="kubelet",resource="runtimeclasses",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kubelet",resource="secrets",subresources="",verb="watch"} 5
node_yurthub_in_flight_requests_collector{client="kubelet",resource="services",subresources="",verb="watch"} 1
node_yurthub_in_flight_requests_collector{client="kubelet",resource="subjectaccessreviews",subresources="",verb="create"} 0
# HELP node_yurthub_in_flight_requests_total total of in flight requests handling by hub agent
# TYPE node_yurthub_in_flight_requests_total gauge
node_yurthub_in_flight_requests_total 18
# HELP node_yurthub_rejected_requests_counter counter of rejected requests for exceeding in flight limit in hub agent
# TYPE node_yurthub_rejected_requests_counter counter
node_yurthub_rejected_requests_counter 0
# HELP node_yurthub_server_healthy_status healthy status of remote servers. 1: healthy, 0: unhealthy
# TYPE node_yurthub_server_healthy_status gauge
node_yurthub_server_healthy_status{server="xx.xx.xx.xx:6443"} 1

Ⅴ. Special notes for reviews

@rambohe-ch rambohe-ch force-pushed the add-metrics-for-yurthub branch from 638c83e to de52ea5 Compare March 18, 2021 10:03
Fei-Guo
Fei-Guo reviewed Mar 19, 2021
View reviewed changes
pkg/yurthub/metrics/metrics.go Outdated Show resolved Hide resolved
pkg/yurthub/metrics/metrics.go Outdated Show resolved Hide resolved
pkg/yurthub/proxy/util/util.go
wrapperRW := newWrapperResponseWriter(w)
start := time.Now()
defer func() {
klog.Infof("%s with status code %d, spent %v", util.ReqString(req), wrapperRW.statusCode, time.Since(start))
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Do we always want to print this log?

Copy link
Member Author

@rambohe-ch rambohe-ch Mar 19, 2021 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

yes, it's easy to trace the request from the log.

@rambohe-ch rambohe-ch force-pushed the add-metrics-for-yurthub branch from de52ea5 to 24f4217 Compare March 19, 2021 09:47
@Fei-Guo Fei-Guo merged commit 989d3f4 into openyurtio:master Mar 19, 2021
zyjhtangtang pushed a commit to zyjhtangtang/openyurt that referenced this pull request Mar 31, 2021
@rambohe-ch @zyjhtangtang
feature: add promtheus metrics for yurthub (openyurtio#238)
a902e5a
@rambohe-ch rambohe-ch deleted the add-metrics-for-yurthub branch May 27, 2021 06:27
MrGirl pushed a commit to MrGirl/openyurt that referenced this pull request Mar 29, 2022
@rambohe-ch
feature: add promtheus metrics for yurthub (openyurtio#238)
335bdfb
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Fei-Guo Fei-Guo Fei-Guo approved these changes

@zyjhtangtang zyjhtangtang Awaiting requested review from zyjhtangtang

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@rambohe-ch @Fei-Guo