Update example config for handling tail-sampling and span metric generation when horizontally scaling collectors

[swar8080]

Hello, we used this page's documentation when setting-up tail-sampling and span metric generation. The guide was helpful but there were a couple things it could've mentioned to make our implementation smoother:

1. The guide currently suggests having two separate deployments of collectors. One just for load-balancing spans and another for processing the load-balanced spans. We started with the two deployments but then found it easier to maintain a single deployment of collectors responsible for both tasks. However, it wasn't obvious (to us at least) that this was possible and I'm guessing others would prefer this approach too.
2. There wasn't any example configuration of how the load-balancing exporter, span metric connector, and tail sampling processor all fit together, so it took some jumping between READMEs and unintentional violations of single-writer assumption (which we weren't aware of) to get it right.

So this documentation suggestion shows how all the components work together in a single collector deployment.

[svrnm]

thanks @swar8080 ! @open-telemetry/collector-approvers PTAL

[jpkrohling]

I like the change, but I'd really prefer to have three collector instances (three config files) instead of one with three pipelines. The reason is that each pipeline (load-balancer, tail-sampler, span metrics) has a different load profile and would scale differently.

You absolutely CAN do it like depicted here, but without understanding the nuances, I'd prefer the official documentation to have each pipeline to be its own deployment.

[jpkrohling]

Suggested change

	- otlp/for_tail_sampling
	- otlp/for_span_metrics

[swar8080]

Hi @jpkrohling, thanks for reviewing

Separating span metrics and tail sampling to different collectors is a good callout. A spike it span volume or span metric cardinality explosion would likely only cause problems for one deployment but not the other. I'll go ahead and change this into three collector configurations, or two if the below persuades you otherwise :).

Maybe we missed the benefits of load balancing being its own deployment. Both set-ups have to receive incoming spans, and load balancing to the same deployment has twice as many receives and exports. But it seems like the extra memory for load balancing is dwarfed by memory needed for tail-sampling and span metrics, which also grows with span volume. For CPU, load balancing exporter used a lot before this optimization, but now our pprof shows it as a small percent of total cpu time. So for us it didn't seem worth the effort of maintaining another deployment, which is why we ended-up consolidating it. That saved us another deployment to monitor, and also one less file to jump between when working on our collector config, since we did some filtering/edits before load balancing. So maybe two separate deployments, each load balancing to themselves, would be a good set-up for a lot of users?