Don't drop instance and job labels in the Prometheus Remote Write Exporter

[rakyll]

This is a follow up to #2897. Even though the instance and job labels are scraped, we were not using them during the export. This change makes sure that the remote write exporter is labelling the outgoing samples with "job" and "instance".

Fixes #575
Fixes #2499
Fixes #2363
Fixes open-telemetry/prometheus-interoperability-spec#37
Fixes open-telemetry/prometheus-interoperability-spec#39

Passing compliance tests:

$ go test --tags=compliance -run "TestRemoteWrite/otelcollector/Job.+" -v ./
=== RUN TestRemoteWrite
=== RUN TestRemoteWrite/otelcollector
=== RUN TestRemoteWrite/otelcollector/JobLabel
=== PAUSE TestRemoteWrite/otelcollector/JobLabel
=== CONT TestRemoteWrite/otelcollector/JobLabel
--- PASS: TestRemoteWrite (10.02s)
--- PASS: TestRemoteWrite/otelcollector (0.00s)
--- PASS: TestRemoteWrite/otelcollector/JobLabel (10.02s)
PASS
ok github.com/prometheus/compliance/remote_write 10.382s
$ go test --tags=compliance -run "TestRemoteWrite/otelcollector/Instance.+" -v ./
=== RUN TestRemoteWrite
=== RUN TestRemoteWrite/otelcollector
=== RUN TestRemoteWrite/otelcollector/InstanceLabel
=== PAUSE TestRemoteWrite/otelcollector/InstanceLabel
=== CONT TestRemoteWrite/otelcollector/InstanceLabel
--- PASS: TestRemoteWrite (10.01s)
--- PASS: TestRemoteWrite/otelcollector (0.00s)
--- PASS: TestRemoteWrite/otelcollector/InstanceLabel (10.01s)
PASS
ok github.com/prometheus/compliance/remote_write 10.291s

[codecov]

Codecov Report

Merging #2979 (62bbea0) into main (d10b842) will increase coverage by 0.00%.
The diff coverage is 100.00%.

@@           Coverage Diff           @@
##             main    #2979   +/-   ##
=======================================
  Coverage   91.65%   91.65%           
=======================================
  Files         312      312           
  Lines       15427    15438   +11     
=======================================
+ Hits        14139    14150   +11     
  Misses        881      881           
  Partials      407      407           

Impacted Files	Coverage Δ
exporter/prometheusremotewriteexporter/exporter.go	89.40% <100.00%> (+0.07%)	⬆️
exporter/prometheusremotewriteexporter/helper.go	99.57% <100.00%> (+0.01%)	⬆️

---

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update d10b842...62bbea0. Read the comment docs.

[rakyll]

cc @bogdandrutu @odeke-em @Aneurysm9 @dashpole

[rakyll]

Contrib tests are failing for Kafka which doesn't seem to be relevant to this change.

[rakyll]

@bogdandrutu TestTraceNoBackend10kSPS|TestTrace1kSPSWithAttrs is failing independent of this change. Please feel free to review.

[etiennejournet]

Hello !

Is this solved ? I still dont have "instance" or "job" shipped to my LogzIo account, but have no idea how to debug it because "loglevel: debug" doesn't give the metrics labels.

My conf for info:

  receivers:
      prometheus:
        config:
          scrape_configs:
          - job_name: k8s
            kubernetes_sd_configs:
            - role: pod
            relabel_configs:
            - source_labels: [__meta_kubernetes_pod_annotation_prometheus_io_scrape]
              regex: "true"
              action: keep
            metric_relabel_configs:
            - source_labels: [__name__]
              regex: "(request_duration_seconds.*|response_duration_seconds.*)"
              action: keep
    service:
      extensions: [health_check]
      pipelines:
        metrics:
          exporters:
          - prometheusremotewrite
          - logging
          receivers:
          - prometheus
    exporters:
      logging:
        logLevel: debug
      prometheusremotewrite:
        endpoint: "https://listener.logz.io:8053"

Thanks,