Allow set custom X-Scope-OrgID header in remote-write of metrics-generator

[luistilingue]

Describe the bug
When using remote-write with a custom X-Scope-OrgID value in metrics-generator, Tempo change the header value to the current one in it.

This code https://github.com/grafana/tempo/blob/main/modules/generator/storage/config_util.go is the responsable of this changing.

To Reproduce
Steps to reproduce the behavior:

1. Start Tempo 1.4.0+ with metrics-generator enabled and storage config with remote-write + custom X-Scope-OrgID header value.
2. Start metric ingestion and check WAL folders - there isn't any with custom header value. The custom header value you've inserted isn't used and the following warning is logged:

caller=config_util.go:30 tenant=tenantA msg="discarding X-Scope-OrgId header" key=X-Scope-OrgID value=TenantAB

Expected behavior
It'll very great have possibility to use a real custom X-Scope-OrgID value due nature of multitenancy and its granularity.

Environment:

• Infrastructure: Kubernetes
• Deployment tool: Helm

Additional Context
Slack conversation: https://grafana.slack.com/archives/C01D981PEE5/p1653420879734839
Configuration:

  metrics_generator:
    ring:
      kvstore:
        store: memberlist
    processor:
      span_metrics:
        histogram_buckets: [0.001, 0.002, 0.004, 0.006, 0.008, 0.010, 0.050, 0.100, 0.200, 0.400, 0.800, 1, 1.4, 2, 5, 10, 15, 20, 40, 100, 500, 1000, 10000]
        dimensions:
          - http.status_code
          - db.system
          - service.version
          - telemetry.auto.version
    storage:
      path: /var/tempo/wal
      wal:
        wal_compression: true
      remote_write:
        - url: http://mimir-staging-distributor.mimir-system.svc.cluster.local:8080/api/v1/push
          headers:
            X-Scope-OrgID: tenantAB
          send_exemplars: true

[yvrhdn]

Agree that Tempo is stricter than necessary. This check was added originally to avoid overwriting the X-Scope-OrgID set internally. Instead I think just logging a warning should be sufficient.

So the behaviour would be:

• if run in single-tenant mode: do not inject X-Scope-OrgID, leave the config alone
• if run in multi-tenant mode: inject the X-Scope-OrgID if not set already; if already set, log a warning that the internal tenant ID will be overwritten

Does this seem reasonable?

[luistilingue]

Yes, it's ok, @kvrhdn .

[rlex]

I just spent good hour trying to figure out what's wrong with my metrics-generator config.

this is my metrics-generator block:

      - metricsGenerator:
          enabled: true
          replicas: {{ .Environment.Values.global_tempo.replicas.metricsGenerator | default "2" }}
          config:
            storage_remote_write:
              - url: http://mimir-distributor-headless.mimir.svc.cluster.local:8080/api/v1/push
                send_exemplars: true
                headers:
                  x-scope-orgid: kube-metrics

According to example in official helm-chart (https://artifacthub.io/packages/helm/grafana/tempo-distributed#activate-metrics-generator) this should work:

    storage_remote_write:
     - url: http://cortex/api/v1/push
       send_exemplars: true
    #   headers:
    #     x-scope-orgid: operations

But it's not and i'm still getting 401 on mimir distributor:

ts=2022-07-05T21:57:13.295741334Z caller=dedupe.go:112 tenant=single-tenant component=remote level=error remote_name=37ddbe url=http://mimir-distributor-headless.mimir.svc.cluster.local:8080/api/v1/push msg="non-recoverable error" count=0 exemplarCount=35 err="server returned HTTP status 401 Unauthorized: no org id"

[yvrhdn]

So what is happening here is that Tempo is removing the x-scope-orgid header you configure in the remote write block. This happens here. We do this to ensure multi-tenant system can not overwrite the tenant ID and write metrics for a different tenant.
You should see a log message from Tempo about this.

In a single-tenant scenario this doesn't make sense and we should remove this check -> #1448 (comment)

I will make a PR to fix this.

As a short-term solution you can put a proxy in between Tempo and Mimir to inject the x-scope-orgid header manully.