Support write_relabel_configs at push_config level

[ponson-thankavel-finastra]

Background: Our K8s cluster is used by multiple tenants (separated by namespaces). We maintain a grafana cloud stack for each of the tenants. We need to collect traces for tenants and stream to respective grafana cloud stack. All tenant applications will send their traces to same endpoint "localhost:55680".

Ask: Is it possible to support write_relabel_configs as part of push_config? below config example illustrates the requirement. grafana agent for metrics & logs support similar configuration.

tempo:
  configs:
    - name: tempo-config
       push_config: 
        - name: tenant1
          basic_auth:
            username: 12345
            password_file: /etc/agent/secrets/tenant1/password
          endpoint: tempo-us-central1.grafana.net
          write_relabel_configs:
            - action: keep
              regex: tenant1-ns1|tenant1-ns2
              source_labels:
                - namespace
        - name: tenant2
          basic_auth:
            username: 45678
            password_file: /etc/agent/secrets/tenant1/password
          endpoint: tempo-us-central1.grafana.net
          write_relabel_configs:
            - action: keep
              regex: tenant2-ns1|tenant2-ns2
              source_labels:
                - namespace
      receivers:
        otlp:
          protocols:
            grpc: 
              endpoint: "localhost:55680"
      scrape_configs:
        - bearer_token_file: /var/run/secrets/kubernetes.io/serviceaccount/token
          job_name: kubernetes-pods
          kubernetes_sd_configs:
            - role: pod

[rfratto]

cc @mapno and @joe-elliott for thoughts here

[mapno]

Hey, thanks for submitting the issue. At the moment, our tracing pipelines do not support a feature that covers all that write_relabel_configs can do in Prometheus. We only support manipulation (i.e. relabelling) of tags on spans that pass through the agent (see docs).

Filtering out spans by attributes/tags is something doable in the agent and could be implemented. However, it's common that not all spans of a trace contain the same tags. Would your use case be covered by filtering by span or would you be interested in filtering by trace?

We're currently working on an implementation of tail-based sampling. This feature will cover filtering traces by tags, but it's probably overkill if the filtering can be done at the span level.

cc/ @ponson-thankavel-finastra

[mapno]

Hi @ponson-thankavel-finastra, we've recently merged an implementation of tail-based sampling to the agent (PR), that allows filtering by trace tags. Let us know if that works for you.

[stale]

This issue has been automatically marked as stale because it has not had any activity in the past 30 days. It will be closed in 7 days if no further activity occurs. Thank you for your contributions.