[Metricbeat] OpenMetrics Module

[sorantis]

OpenMetrics is an effort to standardize exposing metric data based on the Prometheus exposition format.. It's still a draft, and is sandboxed by the CNCF, with the ultimate goal to create an RFC.

Create a dedicated module to monitor any OpenMetrics compliant endpoint using Metricbeat
This can be a light module based on Prometheus.

Checklist

For a metricset to go GA, the following criterias should be met:

* [ ] Supported versions are documented
* [ ] Supported operating systems are documented (if applicable)
* [ ] Integration tests exist

• System tests exist
• Automated checks that all fields are documented
• Documentation
• Fields follow ECS and naming conventions
  * [ ] Dashboards exists (if applicable)
• Kibana Home Tutorial (if applicable)
  • Open PR against Kibana repo with tutorial. Examples can be found here.
• Example data.json exists and an automated way to generate it exists (go test -data)
• Test environment in Docker exist for integration tests

[ChrsMark]

Hey @sorantis !

It looks like it could be easily implemented as a light module on top of prometheus collector metricset (like cockroachdb). The manifest.yml would like like:

default: true
input:
  module: prometheus
  metricset: collector
  defaults:
    metrics_path: /_metrics

In this, I see value in introducing a new option here into the collector metricset so as to have the option to collect only specific metrics. Consider it an equivalent to jmx.amppings we already have in Jolokia module. We have already met it in #15493, #15301 (comment).

cc: @exekias @jsoriano

[jsoriano]

@ChrsMark would this filtering (and the one mentioned in #15493) be covered by processors in light modules (#15923)?

// cc: @mtojek

[ChrsMark]

Technically processors could be used like in https://github.com/elastic/beats/blob/master/x-pack/metricbeat/module/ibmmq/qmgr/manifest.yml.

However in the case of OpenMetrics each case would be different so we need a mechanism to provide the user with, making it possible to define which metrics want or doesn't want to keep from the scrapping endpoint at the configuration level.

So my thought is that script_processor is not so user friendly and maybe we either need to implement another one or just implement the filtering in the Prometheus collector level?

[jsoriano]

However in the case of OpenMetrics each case would be different so we need a mechanism to provide the user with, making it possible to define which metrics want or doesn't want to keep from the scrapping endpoint at the configuration level.

Yes, if we want to avoid scrapping metrics then processors won't work, but I am not sure if exporters in general expose methods to filter metrics when scrapping.

In prometheus we can use a match query to select the metrics to scrape, this works when querying a prometheus server:

- module: prometheus
  period: 10s
  hosts: ["localhost:9090"]
  metrics_path: '/federate'
  query:
    'match[]': '{__name__!=""}'

But I guess this doesn't work with exporters in general.

So my thought is that script_processor is not so user friendly and maybe we either need to implement another one or just implement the filtering in the Prometheus collector level?

I was thinking more in drop_event and drop_fields, that are more straightforward to filter out information. But it is true that to filter out fields based on patters we only have the script processor. We could though extend drop_fields to support patterns.

In general when we have had services that expose a lot of data we have opted by collecting all of it, and telling users to use processors to filter out. This has the advantage of having a common way of filtering out information, while still being able of collecting all the data for interested users. The alternative is to implement different ways of filtering on each module, what is less user friendly (but sometimes way more efficient).

But it is true that prometheus is its own beast. It is completely generic, and sometimes parsing big responses is expensive, specially if we are later going to drop part of the results. So yes, it can be ok to implement some way of mapping. So even if we collect complete responses from exporters we can still decide which ones to parse, and avoid scripts like the one in the IBM MQ module.

I see value in introducing a new option here into the collector metricset so as to have the option to collect only specific metrics. Consider it an equivalent to jmx.mappings we already have in Jolokia module.

@ChrsMark could you make a proposal on how it would look like in the config?

[ChrsMark]

But it is true that prometheus is its own beast. It is completely generic, and sometimes parsing big responses is expensive, specially if we are later going to drop part of the results. So yes, it can be ok to implement some way of mapping. So even if we collect complete responses from exporters we can still decide which ones to parse, and avoid scripts like the one in the IBM MQ module.

👍

I see value in introducing a new option here into the collector metricset so as to have the option to collect only specific metrics. Consider it an equivalent to jmx.mappings we already have in Jolokia module.

@ChrsMark could you make a proposal on how it would look like in the config?

Sure! I will also check how the collector could leverage such an option in an efficient way.

[ChrsMark]

Proposal

Still need to investigate how it could be implemented on the collector's level. Most probably we will have to scape the whole response from the endpoint and filter out the metrics accordingly before sending the events.

Collector config options

1. Include metrics:

include_metrics:
  - ibmmq_*

This would only keep metrics that start with ibmmq_. In the example of ibmmq module this is currently implemented with the script processor.

2. Ignore Metrics

ignore_metrics:
  - process_*
  - go_*
  - istio_mcp_request_acks_total

This would filter out all unnecessary metrics (see the source at istio sample metrics)

3. Metrics

metrics:
  - mixer_config_attributes_total: config_attributes
  - mixer_handler_daemons_total: handler_daemons
 ...

This would implement the mapping which defines which metrics should only be collected and how they should be mapped to fields.

In order to cover cases where we only want to map some specific metrics but we want to keep all the rest as is a * option could be used maybe:

metrics:
  - *
  - mixer_config_attributes_total: config_attributes
  - mixer_handler_daemons_total: handler_daemons

Combining these options

The process of filtering out metrics would be first to apply the include_metrics/ignore_metrics (they are complementary) filters and then on the remaining metrics run the metrics mappings.
None of the above metrics are strictly required. At a first glance the filters could be applied at

beats/metricbeat/module/prometheus/collector/collector.go

Line 83 in 1c32be3

for _, family := range families {

, where we could check if every family should be fetched or not, according to family.Name and the filter lists.

[jsoriano]

@ChrsMark sounds good! I would like to know what @exekias and @sorantis think about this too.
I think you can create a separate issue for this, we could do it independently on how we support open metrics.

Still need to investigate how it could be implemented on the collector's level. Most probably we will have to scape the whole response from the endpoint and filter out the metrics accordingly before sending the events.

Yes, probably, I don't think exporters in general support filtering. Maybe we can do the filtering in GetFamilies() and at least we will save some parsing and the creation of some objects. If it is tricky to do it there, doing it where you mention in the collector sounds good to me too, and we can optimize later if needed.

In order to cover cases where we only want to map some specific metrics but we want to keep all the rest as is a * option could be used maybe.

I would prefer an explicit option for something like this, or we could also not support using both filtering and mapping to avoid tricky cases.

[ChrsMark]

We already had an issue for this, so I moved the proposal there -> #15493

[sorantis]

I see value in introducing a new option here into the collector metricset so as to have the option to collect only specific metrics.

@ChrsMark agreed. this could be a useful feature for users who want to fine tune their metric collection and pull only a subset of exposed metrics. FYI theres' a PR that takes this further and only enables collection of PromQL query results.

[andresrc]

@ChrsMark can we close this one? if not, what's missing?

[ChrsMark]

Updated the PR description with what is missing, however I'm not sure what would be the best option for adding system-tests for this. I'm leaning towards to adding a simple Prometheus instance to scrape from it so as to check mostly that fields are documented.

[ChrsMark]

@andresrc leftovers have been added, we can close this one.