[Kubernetes][Kubebuilder] New Grafana dashboard plugin for visualizing controller-runtime metrics

[nate-double-u]

Your goal is to develop a Kubebuilder Plugin which will generate the manifests required to provide Grafana dashboards for visualizing the default metrics exported. You can begin by first looking into the requirements for exporting the controller-runtime metrics and steps to create a custom dashboard. After figuring out the configuration files which are needed, we can start by creating a plugin when invoked scaffolds them. The plugins here can give a good idea on how they work. An extended goal would be to figure out on how we can make it available on https://grafana.com/grafana/dashboards for users to be able to easily export it.

Mentor(s): Varsha Prasad (@varshaprasad96)

https://github.com/cncf/mentoring/blob/main/summerofcode/2022.md#new-grafana-dashboard-plugin-for-visualizing-controller-runtime-metrics-size-medium

[varshaprasad96]

Following is the summary of questions raised so far. I hope this introduction helps in getting to know the project better.

FAQ

💁‍♀️ What is Kubebuilder? How is it used?

Kubebuilder is often used as a CLI tool that generates projects and provides features to help people create "Operator Solutions".
"Operator solutions" are projects which follow the Operator pattern and extend the K8S API and has the purpose to manage solutions on K8s clusters.

TL'DR:

All Operators are made of controllers, but not all sets of controllers are operators. Operators are sets of controllers that encode human operational knowledge for a specific application (e.g. the Prometheus Operator). Controllers are more general -- for instance, the namespace service account controller isn't an operator because it doesn't have knowledge of a specific application -- it's very general. Kubebuilder helps constructs sets of controllers. Some of those are Operators, some might not be.

Also, Kubebuilder can be used as a lib to help other projects extend their features and/or create nice plugins which can be used as specific helpers. For example, Operator-SDK uses Kubebuilder and scaffolds the same plugins/projects and also provides additional features and others plugins on top of it.

See:

Readme Overview
Readme Motivatios
Who is this for

💡 Examples of its usage:

See https://operatorhub.io/ to have a better idea

❓ Why Operators?

By using operators, it’s not only possible to provide all expected resources but also to manage them dynamically, programmatically, and at execution time. To illustrate this idea, imagine if someone accidentally changed a configuration or removed a resource by mistake; in this case, the operator could fix it without any human intervention.

❓ What is Controller Runtime?

The controller-runtime library provides various abstractions to watch and reconcile resources in a Kubernetes cluster via CRUD (Create, Update, Delete, as well as Get and List in this case) operations. Operators use at least one controller to perform a coherent set of tasks within a cluster, usually through a combination of CRUD operations. It is leveraged by Kubebuilder and Operator SDK. A quick read through on controller-runtime APIs can be found here: https://sdk.operatorframework.io/docs/building-operators/golang/references/client/

🤔 What is the motivation behind this project? Why create a dashboard for metrics?

The controller-runtime library, by default, builds a global prometheus registry and publishes a set of metrics which give insight on the controller’s performance. Any operator which is built using Kubebuilder can publish the same set of metrics and visualize them. It would be helpful if we can provide a plugin that scaffolds the manifests required to visualize these metrics on the grafana dashboard, so that operator authors can benefit from them. An extended goal would be to add this dashboard and make it available in the Grafana dashboards marketplace.

Suggested approach to this problem:

What we need are manifests with which users can wire in their own monitoring stack (grafana and/or prometheus) and observe the metrics. For example, in the current implementation, we create manifests for exposing metrics though kustomize that helps in deploying a service-monitor and service for exposing metrics. With this, users can wire in the service-monitor along with prometheus (from their own monitoring stack) and observe metrics. So the idea is to create a generic solution using the plugin architecture that scaffolds manifests to visualize metrics using grafana.

There are always better ideas and solutions! Please feel free to suggest an alternative in the proposal which you think will make it easier for Kubebuilder users to visualize controller-runtime metrics on grafana dashboards.

💁‍♀️ How to know more about the plugins and how to write them?

See the doc: https://book.kubebuilder.io/plugins/plugins.html

⬆️ A little bit more on plugins! How can I provide the flags for the plugins? Where are they defined?

In the plugin layout, we have some interfaces for the commands like kubebulder init [OPTIONS], kubebulder create API [OPTIONS], kubebulder edit [OPTIONS], kb create webhooks [OPTIONS].
Then, for each plugin, we implement these interfaces (subCommands), e.g.:

• For the declarative plugin which can be called only when we create the project and when we create an api we just implement both subCommands, see: https://github.com/kubernetes-sigs/kubebuilder/tree/master/pkg/plugins/golang/declarative/v1
• However, for the go/v3 plugin, we implement all, see: https://github.com/kubernetes-sigs/kubebuilder/tree/master/pkg/plugins/golang/v3

Then see that we are binding the flags for each subCommand implementation, e.g.

• flags when we run the command kb init using the go/v3 plugin: here
• flags when we run the command kb create api here

🚀 How do I familiarize with the project?

OpenSource projects usually have a Contribution guide, see CONTRIBUTING and also VERSIONING for further details.

Feel free to check the open issues in the project and begin to work and contribute with anything that you find interesting.
Your help and contributions 🥇 are very welcome.

The maintainers actively manage the issues list and try to highlight issues suitable for newcomers (You can filter by good+first+issue also contributions to help on the docs can fit well for the first ones) but you are free to help as it pleases you. The project follows the typical GitHub pull request model. Before starting any work, please either comment on an existing issue or file a new one.

✋ How to get help?

Feel free to ping us and join the #kubebuilder channel in the kubernetes.slack.com slack.
You also can open issues in respective repositories if you find it relevant for others.

👋 More on community meetings

Kubebuilder and Controller-runtime are a part of API Machinery Special Interests Group. The details on community meetings are present here: https://github.com/kubernetes/community/blob/master/sig-list.md

Operator SDK is a part of Operator Framework community. Details on SDK community meetings are here: https://github.com/operator-framework/community#operator-sdk-working-group

Students reached out so far

Slack	Github
Atharva-Shinde
Parul_sahoo	https://github.com/parul5sahoo
Rajiv Singh	https://github.com/iamrajiv
Xin Fu	https://github.com/Kodalien
TianYi(Tony)	https://github.com/Kavinjsir
Avinesh Tripathi	https://github.com/AvineshTripathi

[Kavinjsir]

Hey, @varshaprasad96 , thx for providing the rich context! I also curious about some certain points:

1. When enabling grafana plugins, it seems it depends on the exposure of prometheus metrics. For current kubebuilder, to enabling prometheus metrics, it seems users have to manually uncomment the lines of the config. Are we suppose to enable it automatically if the grafana plugin is used?
2. One further concern based on the above question is, since prometheus stacks is integrated into the common kustomize plugin, why do we want to have an additional plugin for grafana instead of making it also a template under common/kustomize/v1/internal?

[varshaprasad96]

@Kavinjsir Thanks for brining up these questions! Answering them below:

1. Yes, for now Prometheus is optional and is based on uncommenting a few lines in Kustomize files. I would expect Grafana to be the same. By that I mean, Grafana can be enabled only with Prometheus and is optional. If we get inputs from community that they would like it to be enabled by default with Prometheus, then we can change it anytime.
2. This is a good question. It would be nice to have both the options and discuss with the community. Either we can integrate this with the current kustomize plugin, or introduce a new one which combines scaffolding prometheus + Grafana (ie) entire monitoring stack.

[Kavinjsir]

Hi, @varshaprasad96 glad to receive your feedbacks!

I like your idea on 1, this supports a consistent usage of attached resources.
To enable grafana similarly as how current metrics get exposed to prometheus,
guess we are expecting the dashboard to be optionally installed when make deploy?
If so, maybe it is relative to how users deploy their grafana stacks?

• For prometheus-operator, it supports configmap for dashboards installation.
• For grafana/jsonnet, users may use different stack base.(looks like a challenge to fit with kustomize?)
• For Grafana Cloud, users may copy raw json to the ui.

I'm not sure if they may be complex or out of the scope. Or if it is expected to be simple, maybe we can start with providing raw json file, then users can easily copy/paste?

---

I agree with your comments on 2. For me, the good points for integrating grafana with the current kustomize plugin are:

1. It can work in align with prometheus config. (As what you've pointed out in 1.)
2. Ideally, it can be installed during make deploy instead of manual installation.

Alternatively, I am also willing to have a new one (as you suggested in 2):

1. A standalone plugin can leave more space for us to design and implement features.
2. It can be imported/extended externally such as SDK when Kubebuilder is wrapped and used as a library.

Also, I think which option we choose will affect how we can design&implement the things we discuss in 1.