[Tempo] Automatic Logging #551

joe-elliott · 2021-04-20T20:26:01Z

PR Description

Adds an automatic logging processor, documentations, tests and updates the docker-compose example to use it.

Example lines:

span=/cart dur=410000000ns svc=frontend status=STATUS_CODE_UNSET tid=00000000000000002906906e1a662c34
span=/cart dur=461000000ns svc=frontend status=STATUS_CODE_UNSET tid=000000000000000041b1b78d4adc4bd7
span=/product dur=388000000ns svc=frontend status=STATUS_CODE_UNSET tid=0000000000000000b58662a921244284
span=/cart dur=421000000ns svc=frontend status=STATUS_CODE_UNSET tid=00000000000000004e7d51d0834bccf0

Which issue(s) this PR fixes

Notes to the Reviewer

Also updated Grafana, Loki and added Tempo to the default docker-compose example.

PR Checklist

CHANGELOG updated
Documentation added
Tests updated

Signed-off-by: Joe Elliott <number101010@gmail.com>

CLAassistant · 2021-04-20T20:26:06Z

All committers have signed the CLA.

Signed-off-by: Joe Elliott <number101010@gmail.com>

rfratto

Fancy! I have some general concerns about how this interacts with the reloading system.

docs/configuration-reference.md

example/docker-compose/docker-compose.yaml

CHANGELOG.md

rfratto · 2021-04-20T20:39:21Z

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go

+	if lokiInstance == nil {
+		return fmt.Errorf("loki instance %s not found", p.cfg.LokiName)
+	}
+	p.lokiChan = lokiInstance.Promtail().Client().Chan()


This isn't going to work if the promtail restarts 😞 We'll need this to use a channel on the instance (which should be immutable) and have the Loki instance forward from the active promtail over the instance's channel.

Yeah, I did some mild testing and it seemed to work, but my expectation is that in the general case it would not. I was considering:

Shutdown Tempo subsystem

Reload Loki

Reload Tempo

I think this would work.

It'd work, but I'd prefer to try to get this to work with the existing ApplyConfig mechanism and not introducing shutdowns if we can. What were your concerns with forwarding the channel?

@rfratto have a first pass attempt at doing this with the ApplyConfig. Let me know what you think.

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go

pkg/tempo/automaticloggingprocessor/factory.go

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go

Signed-off-by: Joe Elliott <number101010@gmail.com>

pkg/tempo/config.go

Signed-off-by: Joe Elliott <number101010@gmail.com>

mapno

Nice! LGTM 👍

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go

rfratto

Nice! Mostly nits about the docs.

docs/configuration-reference.md

rfratto · 2021-04-22T12:20:13Z

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go

+		return
+	}
+
+	sent := p.lokiInstance.SendEntry(api.Entry{


In the future we'll have to be careful to make sure the Loki instances don't get replaced so this doesn't end up with a stale reference. I think we're fine for now though.

We could be defensive and perform the lookup each time, but I'm not sure it's worth it just yet.

docs/configuration-reference.md

rfratto · 2021-04-22T12:26:08Z

docs/configuration-reference.md

@@ -2045,6 +2045,22 @@ remote_write:
    [ sending_queue: <otlpexporter.sending_queue> ]
    [ retry_on_failure: <otlpexporter.retry_on_failure> ]

+    # automatically log lines to Loki for discovery/metrics
+    automatic_logging:


Just curious: what enables this? Is it defining automatic_logging in the config or is it providing a value for loki_name? Can we document which?

Adding automatic_logging will instantiate the processor, but it will fail with an error message if it can't find the provided loki_name.

docs/configuration-reference.md

Signed-off-by: Joe Elliott <number101010@gmail.com>

docs/configuration-reference.md

Signed-off-by: Joe Elliott <number101010@gmail.com>

* garbage POC, let's goooooooooooOOO! Signed-off-by: Joe Elliott <number101010@gmail.com> * Added basic configuration Signed-off-by: Joe Elliott <number101010@gmail.com> * Added skeleton processor Signed-off-by: Joe Elliott <number101010@gmail.com> * first pass config wiring Signed-off-by: Joe Elliott <number101010@gmail.com> * got things to work? Signed-off-by: Joe Elliott <number101010@gmail.com> * kinda sorta works Signed-off-by: Joe Elliott <number101010@gmail.com> * made things work Signed-off-by: Joe Elliott <number101010@gmail.com> * fixed-ish dur. upped loki and grafana Signed-off-by: Joe Elliott <number101010@gmail.com> * fixed duration, added svc name, did processes correctly Signed-off-by: Joe Elliott <number101010@gmail.com> * Added tempo Signed-off-by: Joe Elliott <number101010@gmail.com> * configurable tags Signed-off-by: Joe Elliott <number101010@gmail.com> * config Signed-off-by: Joe Elliott <number101010@gmail.com> * Addd stopping flag Signed-off-by: Joe Elliott <number101010@gmail.com> * Added tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added configurable stuff Signed-off-by: Joe Elliott <number101010@gmail.com> * Added automatic logging to docs Signed-off-by: Joe Elliott <number101010@gmail.com> * added otel config test Signed-off-by: Joe Elliott <number101010@gmail.com> * Made tid configurable Signed-off-by: Joe Elliott <number101010@gmail.com> * changelog Signed-off-by: Joe Elliott <number101010@gmail.com> * Removed zero valued defaults Signed-off-by: Joe Elliott <number101010@gmail.com> * Put changelog entry in the proper spot Signed-off-by: Joe Elliott <number101010@gmail.com> * - _processor Signed-off-by: Joe Elliott <number101010@gmail.com> * fixed tests Signed-off-by: Joe Elliott <number101010@gmail.com> * Added cross config validation Signed-off-by: Joe Elliott <number101010@gmail.com> * review suggestions Signed-off-by: Joe Elliott <number101010@gmail.com> * First pass SendEntry Signed-off-by: Joe Elliott <number101010@gmail.com> * lint Signed-off-by: Joe Elliott <number101010@gmail.com> * Added configurable timeout Signed-off-by: Joe Elliott <number101010@gmail.com> * Addd logging message when bad things happen Signed-off-by: Joe Elliott <number101010@gmail.com> * Actually check sent :P Signed-off-by: Joe Elliott <number101010@gmail.com> * review cleanup Signed-off-by: Joe Elliott <number101010@gmail.com> * config updates Signed-off-by: Joe Elliott <number101010@gmail.com> * duh Signed-off-by: Joe Elliott <number101010@gmail.com>

joe-elliott added 18 commits April 14, 2021 09:08

garbage POC, let's goooooooooooOOO!

9cfe6d9

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added basic configuration

aed6b14

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added skeleton processor

09d2464

Signed-off-by: Joe Elliott <number101010@gmail.com>

first pass config wiring

6ee0a46

Signed-off-by: Joe Elliott <number101010@gmail.com>

got things to work?

d161788

Signed-off-by: Joe Elliott <number101010@gmail.com>

kinda sorta works

71482e5

Signed-off-by: Joe Elliott <number101010@gmail.com>

made things work

ebc8aa7

Signed-off-by: Joe Elliott <number101010@gmail.com>

fixed-ish dur. upped loki and grafana

1c88fdc

Signed-off-by: Joe Elliott <number101010@gmail.com>

fixed duration, added svc name, did processes correctly

4dc4805

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added tempo

0082076

Signed-off-by: Joe Elliott <number101010@gmail.com>

configurable tags

3ad6059

Signed-off-by: Joe Elliott <number101010@gmail.com>

config

bdda148

Signed-off-by: Joe Elliott <number101010@gmail.com>

Addd stopping flag

74b72a2

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added tests

f6463d9

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added configurable stuff

7d3dc64

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added automatic logging to docs

c974804

Signed-off-by: Joe Elliott <number101010@gmail.com>

added otel config test

e0238ea

Signed-off-by: Joe Elliott <number101010@gmail.com>

Made tid configurable

7e018c3

Signed-off-by: Joe Elliott <number101010@gmail.com>

joe-elliott requested a review from mapno as a code owner April 20, 2021 20:26

joe-elliott added 2 commits April 20, 2021 16:29

changelog

e967715

Signed-off-by: Joe Elliott <number101010@gmail.com>

Merge branch 'main' into loki-exporter

df7002e

rfratto reviewed Apr 20, 2021

View reviewed changes

mapno reviewed Apr 21, 2021

View reviewed changes

pkg/tempo/automaticloggingprocessor/factory.go Outdated Show resolved Hide resolved

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go Show resolved Hide resolved

joe-elliott added 5 commits April 21, 2021 08:27

Removed zero valued defaults

2395de8

Signed-off-by: Joe Elliott <number101010@gmail.com>

Put changelog entry in the proper spot

55d8f4e

Signed-off-by: Joe Elliott <number101010@gmail.com>

- _processor

c210739

Signed-off-by: Joe Elliott <number101010@gmail.com>

fixed tests

7b6761c

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added cross config validation

a8338d5

Signed-off-by: Joe Elliott <number101010@gmail.com>

rfratto reviewed Apr 21, 2021

View reviewed changes

pkg/tempo/config.go Outdated Show resolved Hide resolved

pkg/tempo/config.go Outdated Show resolved Hide resolved

joe-elliott added 6 commits April 21, 2021 15:23

review suggestions

0370820

Signed-off-by: Joe Elliott <number101010@gmail.com>

First pass SendEntry

71c8fd5

Signed-off-by: Joe Elliott <number101010@gmail.com>

lint

a0d6503

Signed-off-by: Joe Elliott <number101010@gmail.com>

Added configurable timeout

f345a73

Signed-off-by: Joe Elliott <number101010@gmail.com>

Addd logging message when bad things happen

dbcefc8

Signed-off-by: Joe Elliott <number101010@gmail.com>

Actually check sent :P

93c3033

Signed-off-by: Joe Elliott <number101010@gmail.com>

mapno approved these changes Apr 22, 2021

View reviewed changes

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go Outdated Show resolved Hide resolved

pkg/tempo/automaticloggingprocessor/automaticloggingprocessor.go Outdated Show resolved Hide resolved

rfratto approved these changes Apr 22, 2021

View reviewed changes

joe-elliott added 2 commits April 22, 2021 08:42

review cleanup

c70d070

Signed-off-by: Joe Elliott <number101010@gmail.com>

config updates

8633639

Signed-off-by: Joe Elliott <number101010@gmail.com>

rfratto reviewed Apr 22, 2021

View reviewed changes

docs/configuration-reference.md Outdated Show resolved Hide resolved

duh

a05e17c

Signed-off-by: Joe Elliott <number101010@gmail.com>

joe-elliott merged commit 238dc73 into grafana:main Apr 22, 2021

mapno mentioned this pull request May 10, 2021

Support FindService && FindOperations grafana/tempo#471

Closed

joe-elliott mentioned this pull request Jul 13, 2021

[Processor] Add new trace logging processor open-telemetry/opentelemetry-collector-contrib#4183

Closed

mattdurham mentioned this pull request Sep 7, 2021

crow doc rfratto/agent#8

Closed

3 tasks

github-actions bot added the frozen-due-to-age Locked due to a period of inactivity. Please open new issues or PRs if more discussion is needed. label Apr 13, 2024

github-actions bot locked as resolved and limited conversation to collaborators Apr 13, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[Tempo] Automatic Logging #551

[Tempo] Automatic Logging #551

joe-elliott commented Apr 20, 2021 •

edited

Loading

CLAassistant commented Apr 20, 2021 •

edited

Loading

rfratto left a comment

rfratto Apr 20, 2021

joe-elliott Apr 21, 2021

rfratto Apr 21, 2021

joe-elliott Apr 21, 2021

mapno left a comment

rfratto left a comment

rfratto Apr 22, 2021

rfratto Apr 22, 2021

joe-elliott Apr 22, 2021

[Tempo] Automatic Logging #551

[Tempo] Automatic Logging #551

Conversation

joe-elliott commented Apr 20, 2021 • edited Loading

PR Description

Which issue(s) this PR fixes

Notes to the Reviewer

PR Checklist

CLAassistant commented Apr 20, 2021 • edited Loading

rfratto left a comment

Choose a reason for hiding this comment

rfratto Apr 20, 2021

Choose a reason for hiding this comment

joe-elliott Apr 21, 2021

Choose a reason for hiding this comment

rfratto Apr 21, 2021

Choose a reason for hiding this comment

joe-elliott Apr 21, 2021

Choose a reason for hiding this comment

mapno left a comment

Choose a reason for hiding this comment

rfratto left a comment

Choose a reason for hiding this comment

rfratto Apr 22, 2021

Choose a reason for hiding this comment

rfratto Apr 22, 2021

Choose a reason for hiding this comment

joe-elliott Apr 22, 2021

Choose a reason for hiding this comment

joe-elliott commented Apr 20, 2021 •

edited

Loading

CLAassistant commented Apr 20, 2021 •

edited

Loading