Two prometheus receivers cannot be defined.

[kahirokunn]

Component(s)

collector

Is your feature request related to a problem? Please describe.

Two prometheus receivers cannot be defined.
The effect of this is that when I use targetAllocator, all static_configs are overwritten and I cannot get cadvisor metrics in EKS as shown below.
When using the targetAllocator, static_configs are automatically rewritten.

receivers:
  prometheus:
    config:
      scrape_configs:
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s

However, there are cases where I want to set static_configs directly.

receivers:
  prometheus:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

If I write it like this, the settings for cadvisor will also be changed.

receivers:
  prometheus:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s

However, defining two receivers like this returns an error.

receivers:
  prometheus/ta:
    config:
      scrape_configs:
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s
  prometheus/static:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

Describe the solution you'd like

I would like to be able to define multiple prometheus receivers.

receivers:
  prometheus/ta:
    config:
      scrape_configs:
      - honor_labels: true
        job_name: otel-collector
        metric_relabel_configs:
        - action: labeldrop
          regex: (id|name)
          replacement: $$1
        - action: labelmap
          regex: label_(.+)
          replacement: $$1
        scrape_interval: 10s
        static_configs:
        - targets:
          - 0.0.0.0:8888
    target_allocator:
      collector_id: ${POD_NAME}
      endpoint: http://otel-targetallocator
      interval: 10s
  prometheus/static:
    config:
      scrape_configs:
      - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
        honor_labels: true
        job_name: kubernetes-nodes-cadvisor
        kubernetes_sd_configs:
        - role: node
        scheme: https
        scrape_interval: 30s
        tls_config:
          ca_file: /var/run/secrets/kubernetes.io/serviceaccount/ca.crt
          insecure_skip_verify: true

Describe alternatives you've considered

No response

Additional context

No response

[swiatekm]

You should be able to get cAdvisor metrics via the Target Allocator this way. What specific issue are you running into? The Target Allocator will take all the scrape configs in your prometheus receiver, generate targets, and allocate them. It shouldn't matter what exactly you have in your scrape config definitions.

We could allow multiple prometheus receivers with the Target Allocator enabled. The reason we haven't is that we've yet to see a compelling use case for it. Generally speaking, there isn't any reason not to use the Target Allocator for everything.

[kahirokunn]

@swiatekm-sumo How can I create a ServiceMonitor for a cadvisor that does not exist as a pod, without using the prometheus-operator? 👀

[swiatekm]

You can't, you need a raw scrape config. The one you've written looks correct to me, and I have a very similar one working with the Target Allocator just fine. Which is why I'd like to know what kind of issue prevents you from using it.