Provide default Grafana dashboards for the produced metrics

[Inithron]

Description

Hi,
I am using Quarkus with the micrometer and some other extensions. Everything works fine but for example the Kafka extension creates a huge amount of metrics. I am new to this topic so I have problems to figure out which of the metrics are really important or valuable. I have found a dashboard visualizing some basic metrics: https://grafana.com/grafana/dashboards/14370-jvm-quarkus-micrometer-metrics which I like very much.

Implementation ideas

My proposal now is to provide similar dashboards for the other extensions: https://quarkus.io/guides/telemetry-micrometer#quarkus-extensions-using-micrometer. This would help everybody visualizing the metrics without spending days in getting familiar with Grafana and creating their own dashboards.

After the creation of this enhancement I have seen that there is a Grafana Dashboard Generator. Unfortunately the generator is not really easy to install/use and the last commit was 4 years ago. So I am not sure if it still works but it gives me another idea: Instead of creating manually Grafana dashboards the dashboard generator could be added to the Quarkus Dev Services or into an own Quarkus extension.

[quarkus-bot]

/cc @ebullient (metrics), @jmartisk (metrics)

[geoand]

cc @alesj @brunobat

[alesj]

We've just added this:

• https://quarkus.io/version/main/guides/observability-devservices

If there is some easy way to pre-add custom / specific dashboard(s) via Grafana configuration, this would be the way to do it.
e.g. adding the right config into the container

• https://github.com/quarkusio/quarkus/blob/main/extensions/observability-devservices/testcontainers/src/main/java/io/quarkus/observability/testcontainers/GrafanaContainer.java

[melloware]

@Inithron i had just asked this same thing on Zulip. I have that same Grafana dashboard as you and I think the goal is to have that in the LGTM stack out of the box.

[Inithron]

@alesj, @melloware I think there was a misunderstanding. My intention was not to add the existing dashboard to the Dev Services. I would like to have for each Quarkus extension one dashboard containing the metrics which are created by this extension. The alternative would be to integrate the dashboard generator into the Dev Services to generate the dashboard(s) automatically.

[melloware]

@Inithron I see so ideally we have dashboards for all the things like Kafka and any other MicroMeter Metrics. Those deashboards should default in the LGTM DevResource so they come out of the box. People can then "export" those dashboards from Grafana to use in their production Grafana environment.

[Inithron]

Yes, that would be great. But first someone needs to create these dashboards.

[melloware]

Yep exactly what I asked on Zulip: https://quarkusio.zulipchat.com/#narrow/stream/187038-dev/topic/Observabilty.20Dev.20Services.20GRAFANA.20OTEL.20LGTM

[melloware]

it doesn't work but here is something of a prototype where we need to add a Prometheus scraper to scrap /q/metrics to get the Micrometer data into dashboard.

public class LgtmContainer extends GrafanaContainer<LgtmContainer, LgtmConfig> {
    protected static final String LGTM_NETWORK_ALIAS = "ltgm.testcontainer.docker";

    protected static final String PROMETHEUS_CONFIG = """
                global:
                  scrape_interval: 15s
                  evaluation_interval: 15s
                storage:
                  tsdb:
                    out_of_order_time_window: 10m
                scrape_configs:
                  - job_name: 'quarkus-micrometer'
                    metrics_path: '/q/metrics'
                    scrape_interval: 15s
                    static_configs:
                      - targets: ['ltgm.testcontainer.docker:8080']
            """;

    public LgtmContainer() {
        this(new LgtmConfigImpl());
    }

    public LgtmContainer(LgtmConfig config) {
        super(config);
        addExposedPorts(config.otlpPort());
        addFileToContainer(PROMETHEUS_CONFIG.getBytes(), "/otel-lgtm/prometheus.yaml");

    }

[melloware]

Ok I got it working. Next I need to figure out how to load the dashboard I want. But making good progress!

[alesj]

Do we need this scraping?
Since isn't OTLP supposed to push this into collector, and the this fwds to Prometheus -> Grafana?

[brunobat]

Correct @alesj. On the grafana devservice there is no scraping. It's all OTLP protocol because it includes an OTel Collector.
The dashboard is usually placed on a specific folder of Grafana, which we have no access to... We can always ask the Grafana folks.
Another solution is to add the dashboard through API or create a dashboard on the Grafana dashboard registry.

[melloware]

@alesj I do not see the micrometer metrics available out of the box with your current LGTM stack. I heard there is something called a "Micrometer Bridge" but i can't find any reference but I can confirm the JVM and Micrometer metrics are not flowing through to LGTM container. Adding the scraper above is the only way I am able to get metrics into my LGTM Grafana Dashboard.

[melloware]

@alesj @brunobat according to this document the Micrometer stuff is disabled by default: https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0

Micrometer metric bridge has been disabled by default. You can enable it using OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true or -Dotel.instrumentation.micrometer.enabled=true.

[brunobat]

@alesj @brunobat according to this document the Micrometer stuff is disabled by default: https://github.com/open-telemetry/opentelemetry-java-instrumentation/releases/tag/v2.0.0

Micrometer metric bridge has been disabled by default. You can enable it using OTEL_INSTRUMENTATION_MICROMETER_ENABLED=true or -Dotel.instrumentation.micrometer.enabled=true.

We don't support that yet. I'm not sure if it works out of the box.

[melloware]

@brunobat @alesj yes that is why i am working on my PR to have a kick button default Quarkus Dashboard. I am going to keep going on it and submit a PR you guys can choose to do what you want with it.

[brunobat]

Sure! Bring it on!

[melloware]

[Inithron]

Hi @melloware,
I have checked #41264 and the screenshot above. But I can't see the metrics for Kafka or other extension. So is my assumption correct, that these metrics are not included in the dashboard?

[melloware]

Correct we decided to go with the simplest metrics possible to start with.

[Inithron]

I appreciate the work which was done so fare. But from a functional point of view this dashboard provides more or less the same information as the dashboard I have linked in this feature request. So the goal, to have dashboards for Kafka and other Quarkus extensions, is not fulfilled. @geoand can you please reopen this feature request?

[geoand]

I will leave that up to the folks that worked on it.

[melloware]

@Inithron PR is welcome. I don't use Kafka so I won't be making a dashboard for it but you seem to have the need so building one and submitting a PR should be easy.

[Inithron]

@melloware you didn't answer the question. I created this feature request to get dashboards for the metrics which are produced by the Quarkus extensions. But you created a dashboard wich exists already. You don't need to create these dashboards, but do you agree to reopen this feature request?

[melloware]

Well I will leave this up to @brunobat where he stated that there are N+ extensions that expose metrics and no one uses every extension so one dashboard wouldn't make sense. So he suggested a separate dashboard per extension in your case Kafka.

So I am not sure one generic ticket of dashboard for all exposed metrics makes sense but I will let @brunobat chine in.

[Inithron]

One generic dashboard makes definitely no sense. But from the beginning on I was talking about dashboards, plural. And if you scroll up and read the initial request again, you will see this is the same what I have suggested:
My proposal now is to provide similar dashboards for the other extensions: https://quarkus.io/guides/telemetry-micrometer#quarkus-extensions-using-micrometer.

And here:

@alesj, @melloware I think there was a misunderstanding. My intention was not to add the existing dashboard to the Dev Services. I would like to have for each Quarkus extension one dashboard containing the metrics which are created by this extension. The alternative would be to integrate the dashboard generator into the Dev Services to generate the dashboard(s) automatically.

[brunobat]

@Inithron thanks for the feedback.
As @melloware mentioned this PR was to provide the minimal setup to have Dashboards up and running.
Please mind you didn't pride details on the contents of the dashboard apart from referencing: https://grafana.com/grafana/dashboards/14370-jvm-quarkus-micrometer-metrics/
This issue was created with follow up improvements: #43599
What you are requesting, a Kafka specific dashboard, makes sense and there are multiple ways to do it:

• Have a section on the dashboard. If we do this for all extensions, the dashboard will be cluttered, but that's not a big problem because this dashboard was intended to be a template that users could use to create their own dashbords... Maybe we can provide a config to load your own dashboard in the devservice?
• Have extension specific dashboards, or get dashboards that provide insights to particular types of apps. A kafka based app, probably doesn't need the HTTP related dashboard components. We can allow people to submit domain specific dashbords that make sense to portions of the Quarkus user base.
• Automatic generation of the dashboard based on automatic generation. This looks appealing but I belied it's out of scope due to it's complexity and the size of our backlog. However, contributions are welcome.

In conclusion. There are many things that can be done in the dashboard area and we need to be pragmatic. Please consider these dashboards a starting point to your own dashboards that eventually can be shared through Quarkus.
Please create a discussion to iron out the final format of the dashboards because this issue is closed and more people could be interested on this subject.

[Inithron]

Hi @brunobat,
For the pull request this is totally ok. But in this feature request I requested default dashboards for all Quarkus extensions. Not only for Kafka. This was only an example because the Kafka extension creates a lot of metrics. But now this feature request was closed but no new dashboard was created. You only recreated the already existing one and added it to the dev services. I like this approach and I appreciate the work you did. But this was something completely different than I have requested. That's why I think this feature request should be reopened, or do you see it different?

[brunobat]

Sorry @Inithron. In my opinion the issue was too vage. We need more concrete steps than "provide similar dashboards for the other extensions: https://quarkus.io/guides/telemetry-micrometer#quarkus-extensions-using-micrometer."

We assumed, Grafana LGTM dev service and provide the basic infra to iterate on the dashboard.
I understand that support metrics for other extensions is needed but we cannot boil the ocean here.

I see this issue as a starting point. Before this, there were not dashboards on the devservices. This is the feature that will allow us to expand them, test them and only then, publish on the Grafana registry.

I'm open to create a new issue, an umbrella issue, where we can detail which metrics and in which extensions we will work next, but the list needs to be concrete in order to prioritise those sub issues. We also need to have a plan on how to deal with the complexity of showing many extensions or if we need multiple dashboards.

[Inithron]

Ok, I understand. But what would be from your point of view the next step? Should I create this "umbrella issue" to decide for wich extension the next dashboard should be created or should I create a discussion to decide if it would be better to create for each extension an own dashboard or to try out the automatic generation of the dashboards?

[brunobat]

I think you should create the discussion first.

[Inithron]

I just wanted to create the discussion but I am not sure in which category I should create it: In Community, Design Discussions, Q&A or should I create a Poll?

[brunobat]

I just wanted to create the discussion but I am not sure in which category I should create it: In Community, Design Discussions, Q&A or should I create a Poll?

Design discussion should be fine

[Inithron]

I have created this design discussion: #44165. So feel free to contribute.