Create Flow components to consume Kubernetes monitoring CRDs

[rfratto]

Today, Grafana Agent Operator must be used if you want to consume ServiceMonitors, PodMonitors, Probes, and PodLogs for collecting telemetry data. I would like to see support for consuming those CRDs added to Grafana Agent itself in the form of Flow components.

Each Flow component will discover a specific CRD, and use the definition of that CRD to collect telemetry data. For example, a prometheus.kubernetes.pod_monitors component collect metrics from pods specified by the set of discovered PodMonitors. Collected metrics are then forwarded to a Prometheus-compatible component, usually prometheus.remote_write.

NOTE: A component to consume PodLogs depends on Promtail components being added to Grafana Agent Flow first.

Adding these components allows Grafana Agent Flow to be a suitable alternative to Grafana Agent Operator, where users would have more power to configure everything Grafana Agent Flow supports instead of being limited to what Grafana Agent Operator happens to expose.

Example pipeline

An example Flow pipeline which uses these components may look like the following:

prometheus.kubernetes.pod_monitors "default" {
  // Look for PodMonitors with a label "application: my-application" in the 
  // default namespace.
  selector {
    labels = {
      application = "my-application"
    } 

    namespace {
      names = ["default"]
    }
  }

  // Send metrics collect from PodMonitor applications to the 
  // specified prometheus.remote_write component. 
  forward_to = [prometheus.remote_write.default.receiver]
}

prometheus.remote_write "default" {
  client {
    url = env("PROMETHEUS_REMOTE_WRITE_URL") 
  }
}

This just serves as an example; the arguments that each of the new components will support is to be determined, though they will likely need to be similar to how you configure a MetricsInstance or LogsInstance with Grafana Agent Operator.

Component names

I haven't been able to come up with a great name for these components which best describe what they do. For now, my preliminary suggestion is to use PROJECT.kubernetes.PLURAL_CRD_NAME for each component, meaning the following components are added:

• prometheus.kubernetes.service_monitors for collecting metrics based on discovered ServiceMonitors
• prometheus.kubernetes.pod_monitors for collecting metrics based on discovered PodMonitors
• prometheus.kubernetes.probes for collecting metrics based on discovered Probes
• loki.kubernetes.pod_logs for collecting logs based on discovered PodLogs.

Benefits

Being able to send metrics from SerivceMonitors, PodMonitors, Probes to any Prometheus-compatible Flow component may be particularly interesting to users, such as being able to convert metrics to OTLP and write them to Kafka:

prometheus.kubernetes.pod_monitors "default" {
  ...

  forward_to = [otelcol.receiver.prometheus.default] 
}

otelcol.receiver.prometheus "default" {
  output {
    metrics = [otelcol.exporter.kafka.default.input] 
  }
} 

// NOTE: this component isn't implemented yet, and just serves 
// as an example for what this approach is capable of doing. 
otelcol.exporter.kafka "default" {
  ... 
}

Tasks

Beta Give feedback

1. Flow: add prometheus.operator.servicemonitors component #3384
   flow frozen-due-to-age +2
   
2. Flow: add prometheus.operator.podmonitors component #3382
   flow frozen-due-to-age +2
3. Flow: add prometheus.operator.probes component #3383
   flow frozen-due-to-age +2
   
4. Flow: add loki.kubernetes.pod_logs component
5. Flow: Support Secrets referenced by Monitoring CRDs. #3749
   flow frozen-due-to-age +2
   

[gloriasc]

@captncraig Any updates on the above mentioned tasklist?