Enterprise Search Stack Monitoring

[kovyrin]

Summary

This PR adds support for Enterprise Searc into the Stack Monitoring plugin. It relies on the new metricbeat module we're shipping in 7.16 (already merged + there is a PR to improve the metricsets) and that will be integrated into the Enterprise Search solution by default (running as a sidecar, controlled via the solution config).

The code in this PR is based primarily on the patterns and style of APM, Beats and Logstash modules and we tried to keep the changes extremely contained to avoid conflicts with any of the de-angularization work that is ongoing within the plugin. The overview page has been built with React (hence the react flag being enabled within the PR, we'll remove it before merging) to align with the new direction for the monitoring plugin.

Our team is planning to support and keep developing this code going forward and we're ready to make whatever changes necessary to align it with the status quo followed by other parts of Stack Monitoring. If any help is needed with testing of the changes, please let us know.

Event Structure

One thing of note in this PR is that Enterprise Search monitoring events fo not have a cluster_uuid field at the root level unlike all other events used by Stack Monitoring. Since the events are generated by metricbeat and elasticsearch metricbeat module has already added a cluster_uuid into the global schema as an alias for their field, we cannot use the same approach and we did not want to add the field to the global schema since it is not compatible with ECS. Instead, we had to change Stack monitoring logic for fetching time series to allow us to pass a flag to it to skip the implicit cluster_uuid filter applied to all queries. You can see the changes in get_metrics.ts and get_series.ts.

Feature Progress

• Enterprise Search is present in the Stack Monitoring UI
  • Displays the metrics we want to ship for the initial release
  • Links to specific details pages
• Basic metrics and stats fetching infrastructure is set up
• React-based page created for the overview page:
  • Overview pane:
    • Total instances
    • Product usage metrics
  • Low-Level metrics pane:
    • HTTP metrics graphs
    • Memory usage graphs
    • Java threads graphs
  • Storage Metrics pane:
    • App Search Engines graph
    • Workplace Search Content Sources graph

Screenshots

Main page

Enterprise Search Overview

Checklist

Delete any items that are not applicable to this PR.

• Any text added follows EUI's writing guidelines, uses sentence case text and includes i18n support
• Documentation was added for features that require explanation or tutorials
• Unit or functional tests were updated or added to match the most common scenarios
• Any UI touched in this PR is usable by keyboard only (learn more about keyboard accessibility)
• Any UI touched in this PR does not create any new axe failures (run axe in browser: FF, Chrome)
• If a plugin configuration key changed, check if it needs to be allowlisted in the cloud and added to the docker list
• This renders correctly on smaller devices using a responsive layout. (You can test this in your browser)
• This was checked for cross-browser compatibility

Risk Matrix

Delete this section if it is not applicable to this PR.

Before closing this PR, invite QA, stakeholders, and other developers to identify risks that should be tested prior to the change/feature release.

When forming the risk matrix, consider some of the following examples and how they may potentially impact the change:

Risk	Probability	Severity	Mitigation/Notes
Multiple Spaces—unexpected behavior in non-default Kibana Space.	Low	High	Integration tests will verify that all features are still supported in non-default Kibana Space and when user switches between spaces.
Multiple nodes—Elasticsearch polling might have race conditions when multiple Kibana nodes are polling for the same tasks.	High	Low	Tasks are idempotent, so executing them multiple times will not result in logical error, but will degrade performance. To test for this case we add plenty of unit tests around this logic and document manual testing procedure.
Code should gracefully handle cases when feature X or plugin Y are disabled.	Medium	High	Unit tests will verify that any feature flag or plugin combination still results in our service operational.
See more potential risk examples

For maintainers

• This was checked for breaking API changes and was labeled appropriately

[phillipb]

Looks pretty good! Small tweaks.

[matschaffer]

Guessing if we address @phillipb 's concerns here we can merge this.

[JasonStoltz]

@elasticmachine merge upstream

[phillipb]

LGTM!

[phillipb]

@elasticmachine merge upstream

[kibana-ci]

💚 Build Succeeded

• Buildkite Build
• Commit: 3b156b8
• Storybooks Preview

Metrics [docs]

Module Count

Fewer modules leads to a faster build time

id	before	after	diff
monitoring	439	445	+6

Async chunks

Total size of all lazy-loaded chunks that will be downloaded as the user navigates the app

id	before	after	diff
monitoring	436.6KB	445.5KB	+8.9KB

Page load bundle

Size of the bundles that are downloaded on every page load. Target size is below 100kb

id	before	after	diff
monitoring	23.6KB	23.6KB	+58.0B

History

• 💚 Build #11071 succeeded 26e5e8b
• 💚 Build #10929 succeeded 3dd0643
• 💔 Build #10795 failed 3d48164
• 💚 Build #10468 succeeded 5244518
• 💔 Build #10446 failed 68f3336
• 💔 Build #10434 failed 3a4951e

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

cc @kovyrin

[kibanamachine]

The following labels were identified as gaps in your version labels and will be added automatically:

• v8.1.0

If any of these should not be on your pull request, please manually remove them.

[kibanamachine]

💚 Backport successful

Status	Branch	Result
✅	8.0

This backport PR will be merged automatically after passing CI.