Prometheus Job and Instance labels

[liamawhite]

When using the Prometheus receiver and exporter I no longer get the job or instance labels that are default in Prometheus. Is this expected behaviour?

When Prometheus scrapes them I lose these fields even with honor_labels: true. I have manually checked the metrics and they are missing these labels so it's not an issue with Prometheus. Is there a way to manually add them back in with some relabel_configs magic? I have tried but it seems to get ignored whenever I use the job or instance labels.

I eventually want two exporters here one of which can be scraped locally by Prometheus and the other will forward onto another collecter. Our default use case is without Prometheus so scraping Prometheus with the collector isn't an option.

Version: v0.2.3

    service:
      pipelines:
        metrics:
          receivers: [prometheus]
          exporters: [prometheus]
    exporters:
      prometheus:
        endpoint: "0.0.0.0:9090"  
    receivers:
      prometheus:
        config:
          global:
            scrape_interval: {{ .Values.scrapeInterval }}
          scrape_configs:
          - job_name: 'pilot'
            kubernetes_sd_configs:
            - role: endpoints
              namespaces:
                names:
                - {{ .Release.Namespace }}
            relabel_configs:
            - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
              action: keep
              regex: istio-pilot;http-monitoring

- job_name: 'oc-agent'
      scrape_interval: 5s
      honor_labels: true
      kubernetes_sd_configs:
      - role: endpoints
        namespaces:
          names:
          - {{ .Release.Namespace }}
      relabel_configs:
      - source_labels: [__meta_kubernetes_service_name, __meta_kubernetes_endpoint_port_name]
        action: keep
        regex: otel-collector;prometheus

[arpitjindal97]

It is so crucial bug and still open from more than 6 months

[hairyhenderson]

I can confirm this is still the current behaviour. I tried setting the job label through a relabel_config, but started getting an error when they didn't match.

The relabel_config I'm using is:

                  - source_labels: [__meta_kubernetes_pod_label_k8s_app]
                    target_label: job

And I'm seeing this error:

{"level":"warn","ts":1602094121.4748123,"caller":"scrape/scrape.go:1058","msg":"Appending scrape report failed","component_kind":"receiver","component_type":"prometheus","component_name":"prometheus","scrape_pool":"etcd-pods","target":"https://10.240.53.137:4001/metrics","err":"unable to find a target group with job=etcd-server"}

When I set the job_name to be the same as what would get relabeled, the error goes away, but I still see neither job nor instance.

In my case, I'll be using Prometheus to scrape the otelcol instance, so I'll just use labels other than job and instance to set them appropriately and then have another relabel_config in Prometheus to set them to what they should be.

I'd like to help fix this, though I'm not sure where to start.

[pingleig]

The error comes from

opentelemetry-collector/receiver/prometheusreceiver/internal/metadata.go

Line 42 in b13bc9d

return nil, errors.New("unable to find a target group with job=" + job)

The problem is the TargetsAll in prometheus scrape manager get its keys from the static prometheus scrape config. It has no way to know the job label within each scraped target, which can be set in the following ways

• exported metrics can contain job label https://prometheus.io/docs/concepts/jobs_instances/
• service discovery can set job label directly
• relabel config based on meta label from service discovery
• in static scrape config as job_name (lowest priority, only used when there is no job label from other places)

However, in otel's prometheus receiver, it can't distinguish the source of job label

opentelemetry-collector/receiver/prometheusreceiver/internal/transaction.go

Line 123 in b13bc9d

job, instance := ls.Get(model.JobLabel), ls.Get(model.InstanceLabel)

One way to workaround as a user is instead of using job label, rename it to job1 and you will pass the prometheus receiver. Then down the pipeline use processors like Metrics Transform Processor to rename is back to job so exporters can the right job label ... (I haven't tested it hough ...

Another way to workaround this is having the receiver figure out a way to have a scrape_job label that won't get override and always match the job_name in scrape_configs. Inside the internal package, check this hidden label and remove it after usage so it won't passed down the pipeline.

[bjrara]

Prometheus receiver needs job and instance label to find the corresponding scraping target and the metric metadata from the cache. The metadata is used to build otel Metrics which would be passed through the pipeline.

How to get the target?

Currently, there's no good way to get the correct target when receiving metrics from Prometheus. The Appender interface doesn't support target information propagation.

Yet the target is known to the appender when it is first initiated. Ref

return newScrapeLoop(
	...
	func(l labels.Labels) labels.Labels { return mutateReportSampleLabels(l, opts.target) },
	func(ctx context.Context) storage.Appender { return appender(app.Appender(ctx), opts.limit) },

It's possible to add target metadata to Appender creation process.

Also there's another open PR (prometheus/prometheus#7771) from Prometheus repo that suggests to pass target metadata into Appender in Add. However, this PR has been stable for 3 months, and I don't know if there would be any follow up actions.

type Appender interface {
	Add(l labels.Labels, m Metadata, t int64, v float64) (uint64, error)
}

Re/label restriction in Prometheus receiver

Today, prometheus receiver could only use job and instance to find the scraping target which puts an extra undocumented restriction on label and relabel.

opentelemetry-collector/receiver/prometheusreceiver/internal/metadata.go

Lines 39 to 48 in b13bc9d

	func (s *metadataService) Get(job, instance string) (MetadataCache, error) {
	targetGroup, ok := s.sm.TargetsAll()[job]
	if !ok {
	return nil, errors.New("unable to find a target group with job=" + job)
	}
	// from the same targetGroup, instance is not going to be duplicated
	for _, target := range targetGroup {
	if target.Labels().Get(model.InstanceLabel) == instance {
	return &mCache{target}, nil

The unable to find a target group error can be triggered by

1. Labeling original metrics with job or instance;
2. Relabeling job or instance in the scrape config to a different value.

Although Prometheus receiver should be able to identify job and instance in case 1 when hornor_label is set to false where label conflicts would be resolved with exported_ prefix, it's hard to distinguish whether it is auto generated by Prometheus or added by users.

Before the issue is fixed, I would suggest people avoid changing these two labels.

[rakyll]

cc @odeke-em

[rakyll]

We will automatically add job and instance labels, this is a known bug. Related to open-telemetry/prometheus-interoperability-spec#7.

[bjrara]

We will automatically add job and instance labels, this is a known bug. Related to open-telemetry/wg-prometheus#7.

I've submitted a fix for auto labeling. #2897. Please let me know if it helps.

[rakyll]

Duplicate of open-telemetry/prometheus-interoperability-spec#37.