[APM] Instrument beat pipeline #17938

jalvz · 2020-04-23T13:55:17Z

What does this PR do?

Instruments beats publishing output and the elasticsearch output.

Why is it important?

To gain insight into performance of libbeat based applications.

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have made corresponding change to the default configuration files
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

How to test this PR locally

Run any beat with ELASTIC_APM_ACTIVE=true set in the environment

Related issues

Supersedes Instrument beat output client #17274

and some basics for the elasticsearch output

also makes transaction end more resilient

jalvz · 2020-04-27T09:13:01Z

Hey, can anyone have a look at this?
Test failures seem unrelated and show up in other PR's too.

axw

LGTM. Added some food-for-thought comments.

axw · 2020-04-30T02:52:44Z

libbeat/publisher/pipeline/output.go

@@ -94,8 +104,7 @@ func (w *clientWorker) run() {
 				continue
 			}
 			w.observer.outBatchSend(len(batch.Events()))
-
-			if err := w.client.Publish(batch); err != nil {
+			if err := w.client.Publish(context.TODO(), batch); err != nil {


Perhaps leave a // TODO comment as well, explaining what needs to be done?

axw · 2020-04-30T03:13:19Z

libbeat/outputs/elasticsearch/client.go

-	data []publisher.Event,
-) ([]publisher.Event, error) {
+func (client *Client) publishEvents(ctx context.Context, data []publisher.Event) ([]publisher.Event, error) {
+	span, ctx := apm.StartSpan(ctx, "publishEvents", "output")


I think it would be better if we didn't create this span, and just relied on the "publish" transaction, but this is probably necessary for now if we really want the labels.

Ideally, the code for recording the labels (events_original, etc.) should be done in a single location, rather than in specific output implementations. This would at lease give high-level information for all outputs. Specific outputs could optionally extend those spans with output-specific information, or add sub-spans (e.g. the apmelasticsearch span).

I think to do that we'd need to change the Batch and/or Observer interfaces (to report batch-specific stats) and/or the Publish signature (to return the stats).

jalvz · 2020-04-30T08:25:34Z

@urso @ycombinator any more comments?
I'd like a final 👍 from either of you before merging

ycombinator · 2020-04-30T09:37:02Z

Hey @jalvz, the libbeat CI failure looks related to this PR? https://travis-ci.org/github/elastic/beats/jobs/681381592#L2209

ycombinator · 2020-04-30T15:43:51Z

One thing that just occurred to me (sorry!) is that we're instrumenting code in the eslegclient package. This package is meant to be a generic Elasticsearch client, which means we shouldn't assume that it's only the libbeat Elasticsearch output or the libbeat Elasticsearch monitoring reporter code is using it. At some point in the future we plan to replace this eslegclient with the official Go Elasticsearch client.

So now I'm wondering if it's a good idea to add instrumentation inside this package or if we should try to instrument around the calls into this package (which would primarily be the Bulk method call).

jalvz · 2020-04-30T15:57:59Z

Ah, I see. I'm happy to move up the the instrumentation if you prefer... that said, without ELASTIC_APM_ACTIVE the tracer is disabled and has no impact. On the other hand, if there are other users of eslegclient outside libbeat that means that they can get instrumentation out of the box for free :)

ycombinator · 2020-04-30T16:08:58Z

Yeah, as things stand today I think the changes in this PR are not really a problem. That's because it looks like the only point of instrumentation (correct me if I'm reading this wrong, btw!) is inside the Bulk method. And that method is only called from either the ES output or the ES monitoring reporter code in libbeat. There are other parts of the Beats codebase that use other eslegclient APIs but not Bulk. And, to your point, this does mean that any code using eslegclient gets APM instrumentation for free which is convenient indeed!

I'm more wondering what this means going forward, when we try to pull out eslegclient (the leg in the name stands for "legacy" , BTW 😉) and replace it with the official Go Elasticsearch client. Would we first need to instrument the latter similarly? Thinking about it more, I don't think this should necessarily block progress on this PR today but perhaps we could simply add a comment near the instrumentation points inside eslegclient about "moving" the instrumentation to the official Go client when the migration happens so we don't lose the instrumentation by accident. Alternatively we could modify this PR to keep the instrumentation around the calls into eslegclient but not inside it (not sure how much that buys us, though?).

WDYT?

jalvz · 2020-04-30T16:25:01Z

Would we first need to instrument the latter similarly?

Yes, exactly. Like, nothing doesn't break or anything if not done, but we would get the value there in doing it. I added a comment about that (great idea).
Moving forward, it would be great to add instrumentation to other parts (mem queue, kafka/logstash outputs, etc), and I think is totally fine that things get moved around and as a result we instrument slightly different things too over time.

ycombinator · 2020-04-30T16:26:23Z

libbeat/esleg/eslegclient/connection.go

@@ -130,6 +130,8 @@ func NewConnection(s ConnectionSettings) (*Connection, error) {
 	}

 	var httpClient esHTTPClient
+	// when dropping the legacy client in favour of the official Go client, it should be instrumented
+	// eg, like in https://github.com/elastic/apm-server/blob/7.7/elasticsearch/client.go


ycombinator

LGTM; merge when CI looks good.

Looking forward to seeing the results of this instrumentation — thanks for doing it!

jalvz · 2020-04-30T17:43:01Z

fantastic, thanks a lot!

jalvz · 2020-05-04T12:27:44Z

Merging this now, thanks all again for review!

This allows beat users to instrument the publishing pipeline by setting ELASTIC_APM_ACTIVE=true in the environment. Co-authored-by: Gil Raphaelli <gil@elastic.co> Co-authored-by: Andrew Wilkins <axwalk@gmail.com>

* [APM] Instrument beat pipeline (#17938) This allows beat users to instrument the publishing pipeline by setting ELASTIC_APM_ACTIVE=true in the environment. Co-authored-by: Gil Raphaelli <gil@elastic.co> Co-authored-by: Andrew Wilkins <axwalk@gmail.com>

graphaelli and others added 18 commits March 26, 2020 18:08

instrument output client

f1f46d6

and some basics for the elasticsearch output

capture publish errors

e915775

also makes transaction end more resilient

clean up some missed tests

5d874d6

Merge branch 'master' into publisher-apm

7f8baae

remove unused Tracer reference

7197c08

create tracer with beat and share with output

a6ac9fa

Merge branch 'master' into publisher-apm

dbc936e

Merge remote-tracking branch 'elastic/master' into publisher-apm

e592c1f

allow the tracer to be nil

4cac58e

add comments

974cdb5

capture reporting errors

0a6eb4f

fix pointer?

7d37861

add transaction and spans tests

ede34e0

Merge remote-tracking branch 'elastic/master' into publisher-apm

2305281

fix bad merge

e48a29b

remove SetTracer

1f0643b

remove other SetTracer and mage fmt

c4c11b1

Merge remote-tracking branch 'elastic/master' into publisher-apm

037830f

jalvz added the enhancement label Apr 23, 2020

jalvz changed the title ~~Publisher apm~~ Instrument beat pipeline with APM Apr 23, 2020

jalvz changed the title ~~Instrument beat pipeline with APM~~ [APM] Instrument beat pipeline Apr 24, 2020

jalvz added 2 commits April 24, 2020 14:13

sync vendor

b29798e

make update

d23b84a

jalvz requested a review from a team April 24, 2020 15:16

jalvz added 3 commits April 24, 2020 17:49

add more context

890617c

Update changelog

f1f66bd

Merge remote-tracking branch 'elastic/master' into publisher-apm

c706d8f

andresrc added [zube]: In Review Team:apm APM Server team issues/PRs labels Apr 27, 2020

jalvz requested a review from axw April 29, 2020 13:51

axw approved these changes Apr 30, 2020

View reviewed changes

jalvz added 2 commits April 30, 2020 10:12

Merge remote-tracking branch 'elastic/master' into publisher-apm

b3a36d1

fix merge

df01669

jalvz added 2 commits April 30, 2020 15:39

fix tests

d79f144

fix test again

e307d6f

more mage fmt

f6c682b

add comment

ce740a4

ycombinator reviewed Apr 30, 2020

View reviewed changes

ycombinator approved these changes Apr 30, 2020

View reviewed changes

jalvz added 2 commits May 4, 2020 12:39

re-add build tag

5bf5bb1

Merge remote-tracking branch 'elastic/master' into publisher-apm

a1c5d3a

jalvz merged commit 2d9ac92 into elastic:master May 4, 2020

zube bot added [zube]: Done and removed [zube]: In Review labels May 4, 2020

jalvz mentioned this pull request May 4, 2020

Cherry-pick #17938 to 7.x: Instrument beat pipeline #18171

Merged

andresrc removed the [zube]: Done label May 6, 2020

This was referenced May 7, 2020

Disable tracer by default #18339

Merged

[APM] Make APM tracer configurable #18360

Closed

bmorelli25 mentioned this pull request Jul 9, 2020

docs: Instrumentation configuration #19765

Merged

6 tasks

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Instrument beat pipeline #17938

[APM] Instrument beat pipeline #17938

jalvz commented Apr 23, 2020 •

edited

Loading

jalvz commented Apr 27, 2020

axw left a comment

axw Apr 30, 2020

axw Apr 30, 2020

jalvz commented Apr 30, 2020 •

edited

Loading

ycombinator commented Apr 30, 2020

ycombinator commented Apr 30, 2020 •

edited

Loading

jalvz commented Apr 30, 2020

ycombinator commented Apr 30, 2020 •

edited

Loading

jalvz commented Apr 30, 2020

ycombinator Apr 30, 2020 •

edited

Loading

ycombinator left a comment

jalvz commented Apr 30, 2020

jalvz commented May 4, 2020

[APM] Instrument beat pipeline #17938

[APM] Instrument beat pipeline #17938

Conversation

jalvz commented Apr 23, 2020 • edited Loading

What does this PR do?

Why is it important?

Checklist

How to test this PR locally

Related issues

jalvz commented Apr 27, 2020

axw left a comment

Choose a reason for hiding this comment

axw Apr 30, 2020

Choose a reason for hiding this comment

axw Apr 30, 2020

Choose a reason for hiding this comment

jalvz commented Apr 30, 2020 • edited Loading

ycombinator commented Apr 30, 2020

ycombinator commented Apr 30, 2020 • edited Loading

jalvz commented Apr 30, 2020

ycombinator commented Apr 30, 2020 • edited Loading

jalvz commented Apr 30, 2020

ycombinator Apr 30, 2020 • edited Loading

Choose a reason for hiding this comment

ycombinator left a comment

Choose a reason for hiding this comment

jalvz commented Apr 30, 2020

jalvz commented May 4, 2020

jalvz commented Apr 23, 2020 •

edited

Loading

jalvz commented Apr 30, 2020 •

edited

Loading

ycombinator commented Apr 30, 2020 •

edited

Loading

ycombinator commented Apr 30, 2020 •

edited

Loading

ycombinator Apr 30, 2020 •

edited

Loading