Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

✨ add metrics for webhooks #234

Merged
merged 1 commit into from
Dec 19, 2018
Merged

✨ add metrics for webhooks #234

merged 1 commit into from
Dec 19, 2018

Conversation

mengqiy
Copy link
Member

@mengqiy mengqiy commented Dec 4, 2018 •
edited
Loading

Add 2 metrics for webhook:

  • Number of total requests
  • Latency of processing admission requests

@k8s-ci-robot k8s-ci-robot added do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. labels Dec 4, 2018
@k8s-ci-robot k8s-ci-robot added the size/M Denotes a PR that changes 30-99 lines, ignoring generated files. label Dec 4, 2018
@k8s-ci-robot k8s-ci-robot added size/L Denotes a PR that changes 100-499 lines, ignoring generated files. and removed size/M Denotes a PR that changes 30-99 lines, ignoring generated files. labels Dec 4, 2018
Liujingfang1
Liujingfang1 reviewed Dec 4, 2018
View reviewed changes
pkg/webhook/admission/webhook.go Outdated
@@ -121,6 +123,8 @@ func (w *Webhook) Handle(ctx context.Context, req atypes.Request) atypes.Respons
case types.WebhookTypeValidating:
resp = w.handleValidating(ctx, req)
default:
metrics.FailedRequests.WithLabelValues(w.Name).Inc()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I have a general question. Maybe I missed that part. It looks that the failure branches will have metrics. How about the successful ones?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is a WIP PR.
I created this PR to see if Travis can reproduce the same strange error I have locally.
I will push more change for the successful ones.

@mengqiy mengqiy force-pushed the webook_metrics branch 2 times, most recently from 1b33798 to 9de9139 Compare December 5, 2018 02:43
DirectXMan12
DirectXMan12 suggested changes Dec 5, 2018
View reviewed changes
pkg/webhook/admission/http.go Outdated
encoder := json.NewEncoder(w)
responseAdmissionReview := v1beta1.AdmissionReview{
Response: response.Response,
}
err := encoder.Encode(responseAdmissionReview)
if err != nil {
log.Error(err, "unable to encode the response")
writeResponse(w, ErrorResponse(http.StatusInternalServerError, err))
metrics.FailedRequests.WithLabelValues(wh.Name).Inc()
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

why are we incrementing both here and right before we call this method?

pkg/webhook/internal/metrics/metrics.go Outdated
// the webhook server has successfully processed.
SucceededRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "controller_runtime_webhook_succeeded_requests",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

this should be a label on requests (like succeeded as the label, and then true and false as the values)

pkg/webhook/internal/metrics/metrics.go Outdated
// the webhook server has received.
TotalRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "controller_runtime_webhook_total_requests",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

requests_total. By convention, all counters end in _total.

pkg/webhook/internal/metrics/metrics.go Outdated
// the webhook server has failed to process with an server internal error.
InternalErrorRequests = prometheus.NewCounterVec(
prometheus.CounterOpts{
Name: "controller_runtime_webhook_internal_error_requests",
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

again, this should be a label on requests_total.

@DirectXMan12
Copy link
Contributor

Please make sure you review https://github.com/kubernetes/community/blob/master/contributors/devel/instrumentation.md and https://prometheus.io/docs/practices/naming/ when contributing metrics.

@mengqiy mengqiy force-pushed the webook_metrics branch 2 times, most recently from fad4014 to ff81845 Compare December 6, 2018 03:31
@mengqiy mengqiy changed the title [WIP] ✨ add metrics for webhooks ✨ add metrics for webhooks Dec 6, 2018
@k8s-ci-robot k8s-ci-robot removed the do-not-merge/work-in-progress Indicates that a PR should not merge because it is a work in progress. label Dec 6, 2018
@mengqiy
Copy link
Member Author

mengqiy commented Dec 6, 2018

PTAL
Hope it makes more sense now.

droot
droot approved these changes Dec 6, 2018
View reviewed changes
Copy link
Contributor

@droot droot left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good to me. Minor suggest to rename.

pkg/webhook/internal/metrics/metrics.go Outdated

// Duration is a prometheus metric which is a histogram of the latency
// of processing an admission request.
Duration = prometheus.NewHistogramVec(
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We should rename it to RequestLatency ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Renamed

@mengqiy
Copy link
Member Author

mengqiy commented Dec 8, 2018

PTAL

@mengqiy
Copy link
Member Author

mengqiy commented Dec 17, 2018

@DirectXMan12 PTAL

@shawn-hurley
Copy link

Maybe this is a follow-on PR, but it would be nice to document in the go doc what metrics are registered when using this package?

@DirectXMan12
Copy link
Contributor

/lgtm /approve

@mengqiy mengqiy mentioned this pull request Dec 19, 2018
Closed
@mengqiy
Copy link
Member Author

mengqiy commented Dec 19, 2018

Bot seems to be dead :/
Applying labels per #234 (comment)

@mengqiy mengqiy added approved Indicates a PR has been approved by an approver from all required OWNERS files. lgtm "Looks good to me", indicates that a PR is ready to be merged. labels Dec 19, 2018
@k8s-ci-robot
Copy link
Contributor

[APPROVALNOTIFIER] This PR is APPROVED

Approval requirements bypassed by manually added approval.

This pull-request has been approved by: mengqiy

The full list of commands accepted by this bot can be found here.

The pull request process is described here

Needs approval from an approver in each of these files:

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

@k8s-ci-robot k8s-ci-robot merged commit 1f7c0c6 into kubernetes-sigs:master Dec 19, 2018
@djzager djzager mentioned this pull request Feb 20, 2020
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@Liujingfang1 Liujingfang1 Liujingfang1 left review comments

@DirectXMan12 DirectXMan12 DirectXMan12 approved these changes

@droot droot droot approved these changes

Assignees
No one assigned
Labels
approved Indicates a PR has been approved by an approver from all required OWNERS files. cncf-cla: yes Indicates the PR's author has signed the CNCF CLA. lgtm "Looks good to me", indicates that a PR is ready to be merged. size/L Denotes a PR that changes 100-499 lines, ignoring generated files.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
6 participants
@mengqiy @DirectXMan12 @shawn-hurley @k8s-ci-robot @droot @Liujingfang1