[APM] Optimize synthtrace #116091

dgieselaar · 2021-10-25T09:04:33Z

Improve performance (10x) of synthtrace by using worker threads and sequential event ids.

TBD:

Update docs

Improve performance of synthtrace by using worker threads and sequential event ids.

…htrace

elasticmachine · 2021-10-25T12:35:35Z

Pinging @elastic/apm-ui (Team:apm)

sorenlouv · 2021-10-25T12:46:39Z

packages/elastic-apm-generator/src/scripts/utils/start_historical_data_upload.ts

+  const limiter = pLimit(workers);
+
+  return Promise.all(new Array(numBatches).fill(undefined).map((_) => limiter(processNextBatch)));


Can we speed this up even more by making it multi-threaded using the cluster module or pm2?

Okay, I now see you are using worker_threads which will add the multi-core support I was alluding to above.

sorenlouv · 2021-10-25T12:59:45Z

packages/elastic-apm-generator/src/scripts/utils/start_historical_data_upload.ts

-        return;
-      }
-      uploadNextBatch();
+    const worker = new Worker(Path.join(__dirname, './upload_next_batch.js'), {


Unfortunate that we have to refer to javascript files. Is something like this possible?

const { Worker, isMainThread } = require("worker_threads"); if (isMainThread) { // start new worker const worker = new Worker(__filename) } else { // run as worker }

we'd have to write everything in JavaScript (supported by node v16) and not use any compiler, I think.

Why would we have to write everything in javascript? The code is executed at runtime where __filename would resolve to the javascript file (even if the source is typescript).

you'd have to build it first though?

sorenlouv

Nice improvement!

sorenlouv · 2021-10-25T21:06:41Z

packages/elastic-apm-generator/src/lib/output/to_elasticsearch_output.ts

    const document = {};
+
+    Object.assign(document, esDocumentDefaults);


nit: what about:

Suggested change

const document = {};

Object.assign(document, esDocumentDefaults);

const document = {

...esDocumentDefaults

};

performance, trying to squeeze as much perf out of this function as possible, as it's called very often.

sorenlouv · 2021-10-25T21:07:57Z

packages/elastic-apm-generator/src/lib/output/to_elasticsearch_output.ts

+    Object.assign(values, event, {
      '@timestamp': new Date(event['@timestamp']!).toISOString(),
      'timestamp.us': event['@timestamp']! * 1000,


nit:

Suggested change

Object.assign(values, event, {

'@timestamp': new Date(event['@timestamp']!).toISOString(),

'timestamp.us': event['@timestamp']! * 1000,

const values = {

...event,

'@timestamp': new Date(event['@timestamp']!).toISOString(),

'timestamp.us': event['@timestamp']! * 1000,

…htrace

kibanamachine · 2021-10-26T10:11:29Z

💚 Build Succeeded

Metrics [docs]

✅ unchanged

History

💔 Build #1350 failed 42d3351
💔 Build #1311 failed aa3a42b

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

kibanamachine · 2021-10-26T10:45:31Z

💔 Backport failed

Status	Branch	Result
❌	7.16	Commit could not be cherrypicked due to conflicts

To backport manually run:
node scripts/backport --pr 116091

…-migrate-away-from-injected-css-js * 'master' of github.com:elastic/kibana: (347 commits) [Upgrade Assistant] Disable UI by default in 8.0 (elastic#115782) [Uptime] Added permission for new tls alert (elastic#116107) [APM] Optimize synthtrace (elastic#116091) Fix ux/apm inspector panel (elastic#116188) [RAC]: add experimental badge to alerts (elastic#116090) Unskip jest handled promise rejections (elastic#116021) [Lens] Improve tick placement for binary formatter (elastic#116158) chore: rename getApmHref to getLegacyApmHref (elastic#115689) [Security Solution] Validate ipv4/CIDR with format x.x.x.x/xx (elastic#116127) [Fleet] Use data stream name in query to get data stream info (elastic#115805) [Uptime] TLS and TLS legacy alert translation mismatch (elastic#116113) New field for integrations field (elastic#116175) Set required to false until the input is not visited (elastic#116099) Enable interactive setup by default (elastic#116141) Add not ready response to interactive setup (elastic#116138) Hide or button if needed (elastic#116124) [ML] Adding datafeed api tests (elastic#116133) Add page title to index advanced page (elastic#116134) chore: rename functions in aggregated_transactions helper (elastic#116001) Fix bug where number rendered as date (elastic#116224) ... # Conflicts: # x-pack/plugins/reporting/server/lib/screenshots/observable.ts # x-pack/plugins/reporting/server/lib/screenshots/open_url.ts

kibanamachine · 2021-10-28T10:43:22Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 116091 or prevent reminders by adding the backport:skip label.

kibanamachine · 2021-10-29T10:53:21Z

Friendly reminder: Looks like this PR hasn’t been backported yet.
To create backports run node scripts/backport --pr 116091 or prevent reminders by adding the backport:skip label.

kibanamachine · 2021-11-01T10:59:48Z

Looks like this PR has a backport PR but it still hasn't been merged. Please merge it ASAP to keep the branches relatively in sync.

dgieselaar added 2 commits October 25, 2021 10:59

[APM] Optimize synthtrace

54775ff

Improve performance of synthtrace by using worker threads and sequential event ids.

Merge branch 'master' of github.com:elastic/kibana into optimize-synt…

aa3a42b

…htrace

dgieselaar added Team:APM All issues that need APM UI Team support release_note:skip Skip the PR/issue when compiling release notes auto-backport Deprecated - use backport:version if exact versions are needed v7.16.0 labels Oct 25, 2021

dgieselaar added 2 commits October 25, 2021 12:52

Update readme

1a3d79e

Fix lint errors

42d3351

dgieselaar marked this pull request as ready for review October 25, 2021 12:35

sorenlouv reviewed Oct 25, 2021

View reviewed changes

sorenlouv approved these changes Oct 25, 2021

View reviewed changes

sorenlouv reviewed Oct 25, 2021

View reviewed changes

Merge branch 'master' of github.com:elastic/kibana into optimize-synt…

e889061

…htrace

dgieselaar merged commit 9d9eeb0 into elastic:master Oct 26, 2021

dgieselaar deleted the optimize-synthtrace branch October 26, 2021 10:42

kibanamachine added the backport missing Added to PRs automatically when the are determined to be missing a backport. label Oct 28, 2021

dgieselaar mentioned this pull request Oct 29, 2021

[7.16] [APM] Optimize synthtrace (#116091) #116761

Merged

dgieselaar added a commit that referenced this pull request Nov 1, 2021

[7.16] [APM] Optimize synthtrace (#116091) (#116761)

b7078f9

kibanamachine removed the backport missing Added to PRs automatically when the are determined to be missing a backport. label Nov 1, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[APM] Optimize synthtrace #116091

[APM] Optimize synthtrace #116091

dgieselaar commented Oct 25, 2021 •

edited

Loading

elasticmachine commented Oct 25, 2021

sorenlouv Oct 25, 2021 •

edited

Loading

sorenlouv Oct 25, 2021

sorenlouv Oct 25, 2021 •

edited

Loading

dgieselaar Oct 25, 2021

sorenlouv Oct 25, 2021

dgieselaar Oct 26, 2021

sorenlouv left a comment

sorenlouv Oct 25, 2021

dgieselaar Oct 26, 2021

sorenlouv Oct 25, 2021

kibanamachine commented Oct 26, 2021

kibanamachine commented Oct 26, 2021

kibanamachine commented Oct 28, 2021

kibanamachine commented Oct 29, 2021

kibanamachine commented Nov 1, 2021

		const limiter = pLimit(workers);

		return Promise.all(new Array(numBatches).fill(undefined).map((_) => limiter(processNextBatch)));

		const document = {};

		Object.assign(document, esDocumentDefaults);

[APM] Optimize synthtrace #116091

[APM] Optimize synthtrace #116091

Conversation

dgieselaar commented Oct 25, 2021 • edited Loading

elasticmachine commented Oct 25, 2021

sorenlouv Oct 25, 2021 • edited Loading

Choose a reason for hiding this comment

sorenlouv Oct 25, 2021

Choose a reason for hiding this comment

sorenlouv Oct 25, 2021 • edited Loading

Choose a reason for hiding this comment

dgieselaar Oct 25, 2021

Choose a reason for hiding this comment

sorenlouv Oct 25, 2021

Choose a reason for hiding this comment

dgieselaar Oct 26, 2021

Choose a reason for hiding this comment

sorenlouv left a comment

Choose a reason for hiding this comment

sorenlouv Oct 25, 2021

Choose a reason for hiding this comment

dgieselaar Oct 26, 2021

Choose a reason for hiding this comment

sorenlouv Oct 25, 2021

Choose a reason for hiding this comment

kibanamachine commented Oct 26, 2021

💚 Build Succeeded

Metrics [docs]

History

kibanamachine commented Oct 26, 2021

💔 Backport failed

kibanamachine commented Oct 28, 2021

kibanamachine commented Oct 29, 2021

kibanamachine commented Nov 1, 2021

dgieselaar commented Oct 25, 2021 •

edited

Loading

sorenlouv Oct 25, 2021 •

edited

Loading

sorenlouv Oct 25, 2021 •

edited

Loading