Add the cluster_uuid field to Enterprise search stats metricset #28287

kovyrin · 2021-10-06T15:58:08Z

What does this PR do?

This PR adds the cluster_uuid field to Enterprise search stats metricset to make it easier to correlate with health metricset events. Additionally, in both metricsets, we have moved the cluster_uuid field to the module level to make it easier to aggregate data across two event types in Stack monitoring and Kibana dashboards.

Why is it important?

This is a follow-up for the recent PR that added the new module and we have just realized that having a uuid in all of our fields would make correlating them in Kibana a lot easier and make those events easier to work with in Kibana monitoring plugin (that relies on cluster UUID to group stack components together).

Checklist

My code follows the style guidelines of this project
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
~~I have made corresponding change to the default configuration files~~
I have added tests that prove my fix is effective or that my feature works
I have added an entry in CHANGELOG.next.asciidoc or CHANGELOG-developer.next.asciidoc.

Author's Checklist

Tested it against the recent Enterprise Search snapshot that has the cluster_uuid field in Stats API responses.

How to test this PR locally

Deploy Enterprise Search
Run metricbeat with the enterprisesearch module enabled
Check stats events produced by the enterprisesearch module and you'll see the new field.

Alternatively, one can use integration tests to make sure the changes work (I've added data tests changes needed).

Related issues

Relates to [Metricbeat] Enterprise Search Module #27549

…ke it easier to correlate with health metricset events

elasticmachine · 2021-10-06T16:37:13Z

💚 Build Succeeded

the below badges are clickable and redirect to their specific view in the CI or DOCS

Expand to view the summary

Build stats

Start Time: 2021-10-13T13:54:06.060+0000
Duration: 107 min 31 sec
Commit: c2a8857

Test stats 🧪

Test	Results
Failed	0
Passed	9504
Skipped	2539
Total	12043

💚 Flaky test report

Tests succeeded.

🤖 GitHub comments

To re-run your PR in the CI, just comment with:

/test : Re-trigger the build.
/package : Generate the packages and run the E2E tests.
/beats-tester : Run the installation tests with beats-tester.

kovyrin · 2021-10-06T18:41:53Z

Integration tests will keep failing until we have a 7.16-SNAPSHOT build of enterprise search with the changes I've made today.

…se events in Stack monitoring

kovyrin · 2021-10-12T23:41:59Z

/test

elasticmachine · 2021-10-13T00:56:32Z

Pinging @elastic/integrations (Team:Integrations)

kovyrin · 2021-10-13T13:45:08Z

metricbeat/docs/fields.asciidoc

@@ -31086,7 +31086,7 @@ type: keyword
 --

 [[exported-fields-enterprisesearch]]
-== enterprisesearch fields
+== Enterprise Search fields


Just noticed we can control this via our module metadata and decided to make it prettier

…evel

kovyrin · 2021-10-13T13:56:45Z

metricbeat/docs/fields.asciidoc

+[float]
+=== product_usage
+
+Aggregate product usage statistics for the Enterprise Search deployment.


Added a few more fields from our stats endpoint to improve the Stack Monitoring experience

kovyrin · 2021-10-13T13:57:16Z

x-pack/metricbeat/module/enterprisesearch/health/_meta/fields.yml

@@ -3,10 +3,6 @@
  release: beta
  description: Enterprise Search health
  fields:
-    - name: cluster_uuid


Moved this to the module level

kovyrin · 2021-10-13T14:24:26Z

Thoroughly tested this while developing the Stack Monitoring integration and it works really well.

richkuz · 2021-10-13T21:10:27Z

👍 New naming changes and metrics look good to me. Nice work @kovyrin !

@sayden would you be willing to review this PR for us, too?

carlosdelest

Good! Moving UUID to top level makes sense, and the additional fields are great for monitoring 👍

kovyrin · 2021-10-14T18:07:24Z

This PR is blocking the testing Kibana team needs to do on the Stack Monitoring integration PR: elastic/kibana#114303 (comment). If we could get this reviewed sooner rather than later, it'd be hugely appreciated 🙇🏻

richkuz · 2021-10-15T14:04:13Z

With two of us having reviewed it in App Search, I feel comfortable with the change. GitHub says we can technically merge now, but we are guests in your house, Beats team :) . @sayden or @ruflin , any objection to us merging this now? We'd like to get a snapshot cooking before Monday if possible.

ruflin · 2021-10-18T09:17:41Z

@masci Can you chime in on who should give a 👍 from your team?

masci · 2021-10-18T15:38:04Z

@richkuz I'm ok merging this even without reviews from my team, @carlosdelest already approved and you folks know better than us.

Feel free to fully own this module, should you have any issue/doubt we'll be happy to help but other than that, no need to include us in the development process.

Thanks for adding more features to the solution!

* Add the cluster_uuid field to Enterprise search stats metricset to make it easier to correlate with health metricset events * Make Enterprise Search module field docs prettier * Updated the docs * Move cluster_uuid to the module level to make it easier to filter those events in Stack monitoring * Add product usage metrics to the stats event * Update mb docs following the addition of new fields * Remove duplicate uuid fields now that we put the uuid at the module level (cherry picked from commit 78bcab3) # Conflicts: # metricbeat/docs/fields.asciidoc

…h stats metricset (#28505) * Add the cluster_uuid field to Enterprise search stats metricset (#28287) * Add the cluster_uuid field to Enterprise search stats metricset to make it easier to correlate with health metricset events * Make Enterprise Search module field docs prettier * Updated the docs * Move cluster_uuid to the module level to make it easier to filter those events in Stack monitoring * Add product usage metrics to the stats event * Update mb docs following the addition of new fields * Remove duplicate uuid fields now that we put the uuid at the module level (cherry picked from commit 78bcab3) # Conflicts: # metricbeat/docs/fields.asciidoc * Fix the conflict Co-authored-by: Oleksiy Kovyrin <oleksiy@kovyrin.net>

…tic#28287) * Add the cluster_uuid field to Enterprise search stats metricset to make it easier to correlate with health metricset events * Make Enterprise Search module field docs prettier * Updated the docs * Move cluster_uuid to the module level to make it easier to filter those events in Stack monitoring * Add product usage metrics to the stats event * Update mb docs following the addition of new fields * Remove duplicate uuid fields now that we put the uuid at the module level

Add the cluster_uuid field to Enterprise search stats metricset to ma…

0e95902

…ke it easier to correlate with health metricset events

kovyrin added enhancement Team:Integrations Label for the Integrations team backport-v7.16.0 Automated backport with mergify labels Oct 6, 2021

kovyrin self-assigned this Oct 6, 2021

botelastic bot added needs_team Indicates that the issue/PR needs a Team:* label and removed needs_team Indicates that the issue/PR needs a Team:* label labels Oct 6, 2021

kovyrin requested review from sayden and carlosdelest October 6, 2021 16:03

Make Enterprise Search module field docs prettier

9fd2d45

Updated the docs

b568ec6

kovyrin added 3 commits October 12, 2021 11:24

Move cluster_uuid to the module level to make it easier to filter tho…

436090a

…se events in Stack monitoring

Add product usage metrics to the stats event

921a5a3

Update mb docs following the addition of new fields

f296f33

kovyrin marked this pull request as ready for review October 13, 2021 00:56

kovyrin commented Oct 13, 2021

View reviewed changes

Remove duplicate uuid fields now that we put the uuid at the module l…

c2a8857

…evel

kovyrin commented Oct 13, 2021

View reviewed changes

kovyrin requested a review from ruflin October 13, 2021 14:23

kovyrin mentioned this pull request Oct 13, 2021

Enterprise Search Stack Monitoring elastic/kibana#114303

Merged

19 tasks

carlosdelest approved these changes Oct 14, 2021

View reviewed changes

kovyrin merged commit 78bcab3 into elastic:master Oct 18, 2021

kovyrin deleted the add-ent-search-stats-cluster-uuid branch October 18, 2021 15:46

mergify bot mentioned this pull request Oct 18, 2021

[7.x](backport #28287) Add the cluster_uuid field to Enterprise search stats metricset #28505

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add the cluster_uuid field to Enterprise search stats metricset #28287

Add the cluster_uuid field to Enterprise search stats metricset #28287

kovyrin commented Oct 6, 2021 •

edited

Loading

elasticmachine commented Oct 6, 2021 •

edited by jenkins-beats-ci bot

Loading

Build stats

Test stats 🧪

kovyrin commented Oct 6, 2021

kovyrin commented Oct 12, 2021

elasticmachine commented Oct 13, 2021

kovyrin Oct 13, 2021

kovyrin Oct 13, 2021

kovyrin Oct 13, 2021

kovyrin commented Oct 13, 2021

richkuz commented Oct 13, 2021

carlosdelest left a comment

kovyrin commented Oct 14, 2021

richkuz commented Oct 15, 2021

ruflin commented Oct 18, 2021

masci commented Oct 18, 2021

Add the cluster_uuid field to Enterprise search stats metricset #28287

Add the cluster_uuid field to Enterprise search stats metricset #28287

Conversation

kovyrin commented Oct 6, 2021 • edited Loading

What does this PR do?

Why is it important?

Checklist

Author's Checklist

How to test this PR locally

Related issues

elasticmachine commented Oct 6, 2021 • edited by jenkins-beats-ci bot Loading

💚 Build Succeeded

Build stats

Test stats 🧪

💚 Flaky test report

🤖 GitHub comments

kovyrin commented Oct 6, 2021

kovyrin commented Oct 12, 2021

elasticmachine commented Oct 13, 2021

kovyrin Oct 13, 2021

Choose a reason for hiding this comment

kovyrin Oct 13, 2021

Choose a reason for hiding this comment

kovyrin Oct 13, 2021

Choose a reason for hiding this comment

kovyrin commented Oct 13, 2021

richkuz commented Oct 13, 2021

carlosdelest left a comment

Choose a reason for hiding this comment

kovyrin commented Oct 14, 2021

richkuz commented Oct 15, 2021

ruflin commented Oct 18, 2021

masci commented Oct 18, 2021

kovyrin commented Oct 6, 2021 •

edited

Loading

elasticmachine commented Oct 6, 2021 •

edited by jenkins-beats-ci bot

Loading