Support mirror traffic pipelines #474

mapno · 2021-03-18T12:42:19Z

PR Description

Adds support for multiple backends in a tracing pipeline. push_config is deprecated in favor of remote_write and batch, which configure the multiple exporters.

remote_write still uses PushConfig, but the idea is to rename it and remove the Batch field once the push_config is removed from the config.

Which issue(s) this PR fixes

Closes #466

Notes to the Reviewer

PR Checklist

CHANGELOG updated
Documentation added
Tests updated

* introduce `remote_write` blocks and `batch` in Tempo config * deprecate `push_config` in Tempo config * support multiple backends for a single tracing pipeline

mapno · 2021-03-18T12:45:21Z

Wasn't really sure how to proceed with all the changes needed in /docs and /production. Also didn't want to go too far updating things before checking that the approach is valid. Let me know what you think.

rfratto

I think this is pretty reasonable but I'm going to tag @joe-elliott for a review.

Does batch need to be applied across all the exporters or is it possible to have (or want) per-endpoint batch settings?

pkg/tempo/config.go

* move Batch field down for clarity regarding its deprecation * remove unnecessary pointer receiver in exporter function * remove redundant check for {remote_write/push_config}.endpoint

Changes config examples to use remote_write/batch instead of push_config

* Sort config pipelines' exporters to make them comparable Maps in Go are not sorted, so the exporters can end up in any order, making the pipeline's exporter slice sorted in that order. If the order is different for actualConfig and expectedConfig, the assertion will fail while both being essentially the same config.

mapno · 2021-03-23T10:09:39Z

Applied the suggested changes and updated the config examples, as well as the config in /production.

joe-elliott

One final thought about backwards compat.

pkg/tempo/config.go

rfratto

Thanks for changing the Tanka stuff too, great work :)

I think my open question from last time got missed. What are the odds someone is going to watch different batch settings per remote_write? Prometheus does this with its queue_config settings. If the same thing doesn't make as much sense for traces, then I think this PR is fine as-is. Otherwise, I think we might want to consider setting up one pipeline per exporter, maybe in a follow-up PR.

(I'm not sure if that actually works though, i.e., if OTel lets you re-use receivers across multiple pipelines.)

pkg/tempo/config.go

rfratto · 2021-03-25T14:23:31Z

pkg/tempo/config.go

+	return exporters, nil
+}
+
+func (c *InstanceConfig) otelConfig() (*configmodels.Config, error) {


Can we log a warning somewhere here if push_config is configured to note that it's deprecated in favor of remote_write?

Added in 53d7f96

mapno · 2021-03-25T16:18:00Z

Thanks for changing the Tanka stuff too, great work :)

I think my open question from last time got missed. What are the odds someone is going to watch different batch settings per remote_write? Prometheus does this with its queue_config settings. If the same thing doesn't make as much sense for traces, then I think this PR is fine as-is. Otherwise, I think we might want to consider setting up one pipeline per exporter, maybe in a follow-up PR.

(I'm not sure if that actually works though, i.e., if OTel lets you re-use receivers across multiple pipelines.)

Yes, sorry I totally missed it before. I'm not sure how often people are going to want per-exporter batch configs, but is something doable. The OTel collector lets you set up pipelines re-using components (i.e. receivers, processors, exporters) as much as you'd like.

I think that different batch configs is most interesting when the exporters are different, which for the agent is not the case (we're always using the otlp exporter). But maybe @joe-elliott can give a better answer since he has more experience operating it.

joe-elliott · 2021-03-30T14:36:29Z

The PR as is covers the necessary cases. The OTEL collector does not allow a batch processor per exporter in a mirrored setup. There can be only one batch exporter for the pipeline that is then mirrored to the two exporters.

Independent pipelines can (and should) have independent batch processors but this is already covered at the []InstanceConfig level.

rfratto · 2021-03-30T14:45:25Z

Thanks for the input! Merging. Thanks again @mapno!

* Support mirroring a trace pipeline to multiple backends * introduce `remote_write` blocks and `batch` in Tempo config * deprecate `push_config` in Tempo config * support multiple backends for a single tracing pipeline * Update CHANGELOG * Apply some review suggestions * move Batch field down for clarity regarding its deprecation * remove unnecessary pointer receiver in exporter function * remove redundant check for {remote_write/push_config}.endpoint * Remove Batch from RemoteWrite config * Update config examples Changes config examples to use remote_write/batch instead of push_config * Fix flaky config test * Sort config pipelines' exporters to make them comparable Maps in Go are not sorted, so the exporters can end up in any order, making the pipeline's exporter slice sorted in that order. If the order is different for actualConfig and expectedConfig, the assertion will fail while both being essentially the same config. * Update /production config to use RemoteWrite * Undeprecate Batch from PushConfig * Add warning log when using push_config

mapno added 2 commits March 18, 2021 13:37

Support mirroring a trace pipeline to multiple backends

50294d1

* introduce `remote_write` blocks and `batch` in Tempo config * deprecate `push_config` in Tempo config * support multiple backends for a single tracing pipeline

Update CHANGELOG

2a86d3f

mapno requested a review from rfratto March 18, 2021 12:47

rfratto reviewed Mar 18, 2021

View reviewed changes

pkg/tempo/config.go Outdated Show resolved Hide resolved

rfratto requested a review from joe-elliott March 18, 2021 14:26

joe-elliott requested changes Mar 18, 2021

View reviewed changes

pkg/tempo/config.go Outdated Show resolved Hide resolved

pkg/tempo/config.go Outdated Show resolved Hide resolved

pkg/tempo/config.go Show resolved Hide resolved

pkg/tempo/config.go Show resolved Hide resolved

Apply some review suggestions

fca5b8b

* move Batch field down for clarity regarding its deprecation * remove unnecessary pointer receiver in exporter function * remove redundant check for {remote_write/push_config}.endpoint

mapno force-pushed the support-mirror-traffic-pipelines branch from b68aa01 to fca5b8b Compare March 18, 2021 19:34

Remove Batch from RemoteWrite config

266d2ad

mapno force-pushed the support-mirror-traffic-pipelines branch from 203101e to 266d2ad Compare March 22, 2021 13:01

mapno added 3 commits March 22, 2021 19:04

Update config examples

646e308

Changes config examples to use remote_write/batch instead of push_config

Update /production config to use RemoteWrite

1aa7b70

joe-elliott reviewed Mar 23, 2021

View reviewed changes

pkg/tempo/config.go Outdated Show resolved Hide resolved

Undeprecate Batch from PushConfig

7dcc416

rfratto approved these changes Mar 25, 2021

View reviewed changes

rfratto requested a review from joe-elliott March 25, 2021 14:29

joe-elliott approved these changes Mar 25, 2021

View reviewed changes

Add warning log when using push_config

53d7f96

mapno force-pushed the support-mirror-traffic-pipelines branch from ca6cfcf to 53d7f96 Compare March 25, 2021 16:21

rfratto merged commit d3c10d9 into grafana:main Mar 30, 2021

mapno deleted the support-mirror-traffic-pipelines branch September 6, 2021 14:47

mattdurham mentioned this pull request Sep 7, 2021

crow doc rfratto/agent#8

Closed

3 tasks

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Apr 9, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 9, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Support mirror traffic pipelines #474

Support mirror traffic pipelines #474

mapno commented Mar 18, 2021 •

edited

Loading

mapno commented Mar 18, 2021 •

edited

Loading

rfratto left a comment

mapno commented Mar 23, 2021

joe-elliott left a comment

rfratto left a comment

rfratto Mar 25, 2021

mapno Mar 25, 2021

mapno commented Mar 25, 2021 •

edited

Loading

joe-elliott commented Mar 30, 2021

rfratto commented Mar 30, 2021

Support mirror traffic pipelines #474

Support mirror traffic pipelines #474

Conversation

mapno commented Mar 18, 2021 • edited Loading

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

mapno commented Mar 18, 2021 • edited Loading

rfratto left a comment

Choose a reason for hiding this comment

mapno commented Mar 23, 2021

joe-elliott left a comment

Choose a reason for hiding this comment

rfratto left a comment

Choose a reason for hiding this comment

rfratto Mar 25, 2021

Choose a reason for hiding this comment

mapno Mar 25, 2021

Choose a reason for hiding this comment

mapno commented Mar 25, 2021 • edited Loading

joe-elliott commented Mar 30, 2021

rfratto commented Mar 30, 2021

mapno commented Mar 18, 2021 •

edited

Loading

mapno commented Mar 18, 2021 •

edited

Loading

mapno commented Mar 25, 2021 •

edited

Loading