Metrics

[randomvariable]

User Story

(copied from kubernetes-sigs/cluster-api-provider-aws#78 )

A cluster operator may want to metrics from the controllers available in their monitoring system of choice, e.g. Prometheus. They would be interested in metrics with regards to the creation, deletion, mutation and failure rates of AWS APIs, and the state of their objects at least to help diagnose when limits have been reached.

Goals

• Produce sets of metrics around operations

Non-Goals

• N/A

Anything else you would like to add:

Current version of controller-runtime should have OpenCensus support.

/kind feature
/priority important/long-term
/milestone next

[k8s-ci-robot]

@randomvariable: You must be a member of the kubernetes-sigs/cluster-api-maintainers GitHub team to set the milestone. If you believe you should be able to issue the /milestone command, please contact your Cluster API Maintainers and have them propose you as an additional delegate for this responsibility.

In response to this:

User Story

(copied from kubernetes-sigs/cluster-api-provider-aws#78 )

A cluster operator may want to metrics from the controllers available in their monitoring system of choice, e.g. Prometheus. They would be interested in metrics with regards to the creation, deletion, mutation and failure rates of AWS APIs, and the state of their objects at least to help diagnose when limits have been reached.

Goals

• Produce sets of metrics around operations

Non-Goals

• N/A

Anything else you would like to add:

Current version of controller-runtime should have OpenCensus support.

/kind feature
/priority important/long-term
/milestone next

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[k8s-ci-robot]

@randomvariable: The label(s) priority/important/long-term cannot be applied. These labels are supported: api-review, community/discussion, community/maintenance, community/question, cuj/build-train-deploy, cuj/multi-user, platform/aws, platform/azure, platform/gcp, platform/minikube, platform/other

In response to this:

User Story

(copied from kubernetes-sigs/cluster-api-provider-aws#78 )

A cluster operator may want to metrics from the controllers available in their monitoring system of choice, e.g. Prometheus. They would be interested in metrics with regards to the creation, deletion, mutation and failure rates of AWS APIs, and the state of their objects at least to help diagnose when limits have been reached.

Goals

• Produce sets of metrics around operations

Non-Goals

• N/A

Anything else you would like to add:

Current version of controller-runtime should have OpenCensus support.

/kind feature
/priority important/long-term
/milestone next

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vincepri]

/priority important-long-term

[k8s-ci-robot]

@vincepri: The label(s) priority/important-long-term cannot be applied. These labels are supported: api-review, community/discussion, community/maintenance, community/question, cuj/build-train-deploy, cuj/multi-user, platform/aws, platform/azure, platform/gcp, platform/minikube, platform/other

In response to this:

/priority important-long-term

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[wfernandes]

Are we thinking of adding prometheus style metrics and an endpoint on the controller manager?

[detiber]

There should already be rudimentary support for the metrics endpoint through controller-runtime, should just need to add additional metrics.

[ncdc]

Yes. We already have the metrics server (

cluster-api/main.go

Line 102 in aa86af2

MetricsBindAddress: metricsAddr,

). +1 to what Jason wrote.

[wfernandes]

Thanks for the tips. I can start to take a look into adding some metrics.

[wfernandes]

/assign

[randomvariable]

Let me know when you start work on it @wfernandes and we'll mark it as lifecycle/active

[wfernandes]

/lifecycle active
Wanted to start a discussion regarding what metrics do we want our controllers to track.

This file (capi-metrics.txt) contains some of the current metrics provided by the controller_runtime framework.

I noticed we have an event recorder that is being used by some of the controllers to track important events. For example, in the machine controller we have FailedDrainNode and FailedDeleteNode events. We could start there for now.

I would like for the additional metrics we expose to be useful to an operator such that they may want to set up an alerting rule based on these metrics for example.

[wfernandes]

Adding these for reference:
Kubernetes Instrumentation Guidelines: https://github.com/kubernetes/community/blob/master/contributors/devel/sig-instrumentation/instrumentation.md

[detiber]

We've talked about adding additional events to the Machine. For example, I think it would make sense to emit an event based on the following things:

• Status.BootstrapReady is set
• Status.InfrastructureReady is set
• Status.NodeRef is set
• ErrorReason/ErrorMessage is set

[randomvariable]

We've also just added https://github.com/kubernetes-sigs/cluster-api-provider-aws/blob/master/docs/observability.md to the AWS provider, a similar doc here would also be useful to consumers.

[harishspqr]

It would be great if the cluster/machine deployment create/delete/resize latencies can be included in the CRD status field once they are created.

[ncdc]

@harishspqr if we record them, they wouldn't be included in the status field. They would be exposed as metrics accessible from the metrics server port.

[detiber]

@harishspqr What type of calculation are you looking for out of these latencies?

• The duration from request to completion?
• The duration from request to start of provisioning?
• The duration from start of provisioning to finished provisioning?
• All of the above?

[fejta-bot]

Issues go stale after 90d of inactivity.
Mark the issue as fresh with /remove-lifecycle stale.
Stale issues rot after an additional 30d of inactivity and eventually close.

If this issue is safe to close now please do so with /close.

Send feedback to sig-testing, kubernetes/test-infra and/or fejta.
/lifecycle stale

[vincepri]

/lifecycle frozen

[sbueringer]

/reopen
/remove-lifecycle rotten

[k8s-ci-robot]

@sbueringer: Reopened this issue.

In response to this:

/reopen
/remove-lifecycle rotten

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[vincepri]

@fabriziopandini to split this issue into multiple ones and provide additional context / resolution

[fabriziopandini]

I think the best way forward for the metrics effort is to join forces with https://github.com/mercedes-benz/cluster-api-state-metrics in order to provide a first set of "core metrics" about the Cluster API API objects.
What I really like about this approach is that metrics are a pluggable component, running out of the CAPI/provider process, providing not only valuable insights but also a first validation about the fact that our APIs provides all the info that an operator is looking for

WRT to implementing metrics inside CAPI, I will limit this to a limited number of use cases TBD and focused on internal functioning of the controllers, e.g. :

• counter of errors while connecting remote Clusters
• average duration of runtime extension calls
• etc.

I will raise the point at the next office hours, to see if there is agreement on this course of actions

[enxebre]

WRT to implementing metrics inside CAPI, I will limit this to a limited number of use cases TBD and focused on internal functioning of the controllers, e.g. :
counter of errors while connecting remote Clusters
average duration of runtime extension calls
etc.

This make sense to me, this can be used as basic SLIs that e.g CI systems consume to ensure SLOs are met.

Regarding https://github.com/mercedes-benz/cluster-api-state-metrics which looks awesome, in order to move it towards a community pluggable component solution I think It'd be good to have a common doc/issue to discuss and agree on all use cases/goals we want to achieve with those metrics.

[vincepri]

We could also ask the maintainers of cluster-api-state-metrics if they might be interested in donating the project to CNCF? We could then incorporate it within Cluster API and ship it as an extra component

[sbueringer]

If I understood correctly, maintainers offered that already :) (@chrischdi @tobiasgiese please confirm :))

@vincepri What would be the road towards donation?

[chrischdi]

Yes that was one possible target or reason for us to even publish it to github.

I think the donation would help to get more contributions for the project and would increase acceptance to use it.

We try to talk internally to the people to clarify if proceeding is okay for them (to accomplish the formalities) 👍

[chrischdi]

So after having a first talk, we would be willed to donate the project :-)

Would a donation to the kubernetes-sig be a better fit than donating it to the CNCF?

If contributing to the kubernetes-sig the requirements are documented here: source

Which would require to

• relicense the code to Apache 2.0 (should be also okay from our side)
• ensure all current Contributors have signed the CNCF CLA (currently the case)
• ensure license of dependencies are acceptable (should be the case)
• adopt the CNCF CLA bot, merge bot and Kubernetes PR commands/bots
• All OWNERS of the project must also be active SIG members (currently me and @tobiasgiese, both active at cluster-api-provider-openstack as maintainers)
• Must be approved by the process spelled out in the SIG's charter and a publicly linkable written decision should be available for the same.
• SIG must already have identified all of their existing subprojects and code, with valid OWNERS files, in sigs.yaml

[fabriziopandini]

@chrischdi that's great news!
let's discuss the next steps during office hours!

Personally I'm also intrigued by Vince's suggestion

We could then incorporate it within Cluster API and ship it as an extra component

Because this can help in having a quicker start (it is just a PR), and most importantly it allows a stricter collaboration with the CAPI core team while migrating the project to v1beta1, consolidating the first set of metrics etc. but I will be happy with both options

[sbueringer]

I agree, it's an interesting option to integrate it in the core CAPI repo. It would definitely make it a lot easier to integrate it in Tilt, the release process, ... etc.

I think in that case we would create a new (probably top-level) folder in the CAPI repo with an OWNER file and would add the current maintainers as approvers there.

From a high-level perspective, it would essentially be a core component of ClusterAPI.

[chrischdi]

That sounds good. I will attend next week 👍 .

@tobiasgiese or @johannesfrey : would be great if one of you could also free up some time :-)

[sbueringer]

One additional point:

I think it also helps us to close the feedback loop for CAPI developers for metrics better. Just like we currently try to make logging part of our development workflow.

If we're able to include metrics and logging in our development workflows (especially for debugging) it will help to continuously improve both and we will slowly considering them as part of the ClusterAPI "deliverable". Essentially take on responsibility not only for the behavior/implementation of our controllers but also about how observable/ operable / debuggable they are.

I think this helps us to get closer to the principle, that usually things improve once the folks responsible for development are using them themselves and rely on them (in it's most extreme form if they are on call for it, but in our case if we integrate them into dev workflows).

[tobiasgiese]

@tobiasgiese or @johannesfrey : would be great if one of you could also free up some time :-)

Sure, I'll attend also, looking forward to it

[johannesfrey]

@tobiasgiese or @johannesfrey : would be great if one of you could also free up some time :-)

Sure, I'll attend also, looking forward to it

Sure, count me in as well :).

[chrischdi]

First of all thanks for the discussion in the office hours.

As follow-up: so the next steps would be:

• to add a small proposal about the feature set and wanted metrics which then targets to
• integrate it as a next step into https://github.com/kubernetes-sigs/cluster-api/tree/main/exp via a PR

[tobiasgiese]

We created a first draft of the Cluster API State Metrics proposal
Thanks @chrischdi and @johannesfrey 🙂🎉

[killianmuldoon]

If there's no objections I think we should close this issue now that #6404 is open. For future conversations around metrics it's better IMO to have issue with a tighter scope.

[fabriziopandini]

+1 to close this as soon and #6404 merges

[sbueringer]

I think #6404 only addresses state metrics. This still leaves metrics exposed by our controllers. Did I understand correctly that we would open a new tightly scoped issue for this before closing this issue?

[killianmuldoon]

@sbueringer I think this issue is very old and has gone through too many transformations to be really useful anymore so any future work should go in newer more scoped issues.

I'm not sure exactly what specific metrics exposed by controllers we'd be missing by closing this issue now though.

[fabriziopandini]

/close
we can open new issue as soon as we have actionable items for metrics exposed by our controllers

[k8s-ci-robot]

@fabriziopandini: Closing this issue.

In response to this:

/close
we can open new issue as soon as we have actionable items for metrics exposed by our controllers

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.