Deploying Tempo distributors to each cluster and let them connect to same ingester

[evan361425]

Which architecture is recommended?

1. Multiple cluster and each has distributors only

flowchart TD
  subgraph c1 [Cluster 1]
    dist1[Distributor]
    ing[Ingester]
    other[All other components]
    ing -.- other
    a1[App 1] --> dist1
    a2[App 2] --> dist1
  end
  subgraph c2 [Cluster 2]
    dist2[Distributor]
    a3[App 3] --> dist2
    a4[App 4] --> dist2
  end
  dist1 --> ing
  dist2 --> ing

Loading

2. Multiple cluster and each has all components

flowchart TD
  subgraph c1 [Cluster 1]
    dist1[Distributor]
    ing1[Ingester]
    ing1 -.- other1[All other components]
    a1[App 1] --> dist1
    a2[App 2] --> dist1
    dist1 --> ing1
  end
  subgraph c2 [Cluster 2]
    dist2[Distributor]
    ing2[Ingester]
    a3[App 3] --> dist2
    a4[App 4] --> dist2
    dist2 --> ing2
    ing2 -.- other2[All other components]
  end

Loading

---

Method 1 has better resource usage and decrease complexity but not sure if I miss any considerations.
Do Tempo has any official recommendations?

[joe-elliott]

Can you explain you goals? It's hard to give advice w/o understanding what you are trying to achieve.

Multiple cluster and each has distributors only

This just looks like one Tempo cluster to me. What does "cluster 1" and "cluster 2" mean?

[evan361425]

Thank you for your response. Currently, we are preparing to deploy Tempo in two K8s clusters, which represent different regions. If both clusters share the same querier, we can point Grafana to the same datasource. However, we are not sure if this is a good practice because we would have to expose the ingester and metrics generator of cluster 1.

Additionally, if it is recommended to keep the two clusters independent, could we let the two different sets of Tempo share the same object storage?

[joe-elliott]

However, we are not sure if this is a good practice because we would have to expose the ingester and metrics generator of cluster 1.

This is a networking security decision unrelated to Tempo. I can't really help you here.

Additionally, if it is recommended to keep the two clusters independent, could we let the two different sets of Tempo share the same object storage?

With some advanced tweaks I believe this is technically possible, but I've never run Tempo this way so I can't be sure something would break unexpectedly. I would not advise it.

My biggest concern about splitting a tempo cluster across k8s clusters is that networking latency would be higher between clusters which would impact the memberlist ring propagation. You can certainly try.

[evan361425]

@joe-elliott Thanks for the reply! Considering the challenges of using distributed Tempo in a multi-cluster environment, we decided to use the OpenTelemetry Collector to support multi-cluster setups without managing multiple S3 endpoints.

It provide several features:

• Error collection by tail-sampling-processor in single cluster;
• Load balancing by load-balancing-exporter allow scaling the collector.

flowchart TD
  subgraph Cluster1
    lb1[otel-lb] --100%--> col[otel-collector]
    col --sampled--> tempo[Tempo]
  end
  subgraph Cluster2
    lb2[otel-lb]
  end
  subgraph Cluster3
    lb3[otel-lb]
  end
  lb2 --100%--> col
  lb3 --100%--> col

Loading

Both otel-lb and otel-collector are OpenTelemetry Collector, which can easily scale up.

[joe-elliott]

Very nice! This looks like it will work quite well.