Skip to content

Commit

Permalink
Enterprise Search Stack Monitoring (#114303)
Browse files Browse the repository at this point in the history
* Added enterprise search panel, corrected queries

* Update the index pattern for Enterprise Search

* Typescript error ignore

* Our timestamp fields are called @timestamp (per ECS)

* Adjust Enterprise Search index patterns with the rest of monitoring plugin patterns (including CCS, etc)

* Initial implementation of the Enterprise Search overview panel (health only)

* Add a basic stub for enterprise search response fields

* Cleanup aggs configs

* Bring back a file deleted by mistake

* Started working on the overview page

* Correctly use heap_max as the total heap

* Ent search breadcrumbs

* Simple overview

* Allow the cluster_uuid filter to be skipped while fetching metrics

* Cleanup

* Switch to module-level uuid field and use both types of events

* Add stats-based product usage metrics + apply filter paths to reduce traffic

* Change the name of the ent search overview class

* Move the standalone cluster hack in the the internal function

* Change the overview page to show product usage metrics + introduce enterprise search stats in addition to metrics (they are fetched differently and allow us to reuse the stats code we have for the main page panel)

* Cluster UUID is at the module level now

* Simplify ent search pages structure, only have one overview page

* Fix ent search icon

* Add total instances

* Product usage metric graphs

* Simplify metrics loading in the overview page since we load all metrics anyways

* Add more enterprise search overview metrics

* Avoid duplicate labels

* linting

* Revert "Simplify metrics loading in the overview page since we load all metrics anyways"

This reverts commit 4bd67ab.

* Switch to multiple timeseries per graph

* Reorder graphs and metrics for better experience

* Typescript fixes

* i18n fixes

* Added a couple more JVM metrics

* Completely covered JVM metrics

* Convert Enterprise Search component to Typescript

* Switch config setting back

* Remove the nodes link since it raises more questions than it solves

* Update jest snapshots with the new metrics

* Remove console statement

* Properly handle cases when aggregations return no data for Enterprise Search

* Add a functional test for the Enterprise search cluster list panel

* Add a functional test for Enterprise Search overview page

* Update multicluster API response fixture with the new enterprise search response key

* Default uptime value is 0

* update overview fixture

* More fixture updates

* Remove fixmes

* Fix imports

* Properly export type

* Maybe fix the type checking error

* PR Feedback

* TS fixes

Co-authored-by: cdelgado <carlos.delgado@elastic.co>
Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
Co-authored-by: Jason Stoltzfus <jason.stoltzfus@elastic.co>
  • Loading branch information
4 people authored Dec 7, 2021
1 parent fcd47f3 commit 7929123
Show file tree
Hide file tree
Showing 47 changed files with 37,019 additions and 524 deletions.
8 changes: 8 additions & 0 deletions x-pack/plugins/monitoring/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ export const INDEX_PATTERN_LOGSTASH = '.monitoring-logstash-6-*,.monitoring-logs
export const INDEX_PATTERN_BEATS = '.monitoring-beats-6-*,.monitoring-beats-7-*';
export const INDEX_ALERTS = '.monitoring-alerts-6*,.monitoring-alerts-7*';
export const INDEX_PATTERN_ELASTICSEARCH = '.monitoring-es-6-*,.monitoring-es-7-*';
export const INDEX_PATTERN_ENTERPRISE_SEARCH = '.monitoring-ent-search-*';

// This is the unique token that exists in monitoring indices collected by metricbeat
export const METRICBEAT_INDEX_NAME_UNIQUE_TOKEN = '-mb-';
Expand Down Expand Up @@ -158,6 +159,7 @@ export const CODE_PATH_LOGSTASH = 'logstash';
export const CODE_PATH_APM = 'apm';
export const CODE_PATH_LICENSE = 'license';
export const CODE_PATH_LOGS = 'logs';
export const CODE_PATH_ENTERPRISE_SEARCH = 'enterprise_search';

/**
* The header sent by telemetry service when hitting Elasticsearch to identify query source
Expand All @@ -177,6 +179,12 @@ export const KIBANA_SYSTEM_ID = 'kibana';
*/
export const BEATS_SYSTEM_ID = 'beats';

/**
* The name of the Enterprise Search System ID used to publish and look up Enterprise Search stats through the Monitoring system.
* @type {string}
*/
export const ENTERPRISE_SEARCH_SYSTEM_ID = 'enterprise_search';

/**
* The name of the Apm System ID used to publish and look up Apm stats through the Monitoring system.
* @type {string}
Expand Down
3 changes: 3 additions & 0 deletions x-pack/plugins/monitoring/common/types/es.ts
Original file line number Diff line number Diff line change
Expand Up @@ -677,6 +677,9 @@ export interface ElasticsearchMetricbeatSource {
};
};
};
enterprisesearch?: {
cluster_uuid?: string;
};
}

export type ElasticsearchSource = ElasticsearchLegacySource & ElasticsearchMetricbeatSource;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,16 @@ function getApmBreadcrumbs(mainInstance: any) {
return breadcrumbs;
}

// generate Enterprise Search breadcrumbs
function getEnterpriseSearchBreadcrumbs(mainInstance: any) {
const entSearchLabel = i18n.translate('xpack.monitoring.breadcrumbs.entSearchLabel', {
defaultMessage: 'Enterprise Search',
});
const breadcrumbs = [];
breadcrumbs.push(createCrumb('#/enterprise_search', entSearchLabel));
return breadcrumbs;
}

function buildBreadcrumbs(clusterName: string, mainInstance?: any | null) {
const homeCrumb = i18n.translate('xpack.monitoring.breadcrumbs.clustersLabel', {
defaultMessage: 'Clusters',
Expand All @@ -212,6 +222,9 @@ function buildBreadcrumbs(clusterName: string, mainInstance?: any | null) {
if (mainInstance?.inApm) {
breadcrumbs = breadcrumbs.concat(getApmBreadcrumbs(mainInstance));
}
if (mainInstance?.inEnterpriseSearch) {
breadcrumbs = breadcrumbs.concat(getEnterpriseSearchBreadcrumbs(mainInstance));
}

return breadcrumbs;
}
Expand Down
9 changes: 9 additions & 0 deletions x-pack/plugins/monitoring/public/application/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {
CODE_PATH_KIBANA,
CODE_PATH_LOGSTASH,
CODE_PATH_APM,
CODE_PATH_ENTERPRISE_SEARCH,
} from '../../common/constants';
import { BeatsInstancePage } from './pages/beats/instance';
import { ApmOverviewPage, ApmInstancesPage, ApmInstancePage } from './pages/apm';
Expand All @@ -43,6 +44,7 @@ import { ElasticsearchMLJobsPage } from './pages/elasticsearch/ml_jobs_page';
import { ElasticsearchNodeAdvancedPage } from './pages/elasticsearch/node_advanced_page';
import { ElasticsearchCcrPage } from './pages/elasticsearch/ccr_page';
import { ElasticsearchCcrShardPage } from './pages/elasticsearch/ccr_shard_page';
import { EntSearchOverviewPage } from './pages/enterprise_search/overview';
import { MonitoringTimeContainer } from './hooks/use_monitoring_time';
import { BreadcrumbContainer } from './hooks/use_breadcrumbs';
import { HeaderActionMenuContext } from './contexts/header_action_menu_context';
Expand Down Expand Up @@ -312,6 +314,13 @@ const MonitoringApp: React.FC<{
fetchAllClusters={false}
/>

<RouteInit
path="/enterprise_search"
component={EntSearchOverviewPage}
codePaths={[CODE_PATH_ENTERPRISE_SEARCH]}
fetchAllClusters={false}
/>

<Redirect
to={{
pathname: '/loading',
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React from 'react';
import { PageTemplate } from '../page_template';
import { PageTemplateProps } from '../page_template';

type EntSearchTemplateProps = PageTemplateProps;

export const EntSearchTemplate: React.FC<EntSearchTemplateProps> = (props) => {
return <PageTemplate {...props} product="enterprise_search" />;
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,76 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import React, { useContext, useState, useCallback, useEffect } from 'react';
import { i18n } from '@kbn/i18n';
import { find } from 'lodash';
import { ComponentProps } from '../../route_init';
import { EntSearchTemplate } from './ent_search_template';
import { GlobalStateContext } from '../../contexts/global_state_context';
import { useCharts } from '../../hooks/use_charts';
import { useKibana } from '../../../../../../../src/plugins/kibana_react/public';
import { EnterpriseSearchOverview } from '../../../components/enterprise_search/overview';
import { BreadcrumbContainer } from '../../hooks/use_breadcrumbs';

export const EntSearchOverviewPage: React.FC<ComponentProps> = ({ clusters }) => {
const globalState = useContext(GlobalStateContext);
const { zoomInfo, onBrush } = useCharts();
const { services } = useKibana<{ data: any }>();
const clusterUuid = globalState.cluster_uuid;
const ccs = globalState.ccs;
const { generate: generateBreadcrumbs } = useContext(BreadcrumbContainer.Context);
const cluster = find(clusters, {
cluster_uuid: clusterUuid,
}) as any;

const [data, setData] = useState(null);

const title = i18n.translate('xpack.monitoring.entSearch.overview.routeTitle', {
defaultMessage: 'Enterprise Search - Overview',
});

const pageTitle = i18n.translate('xpack.monitoring.entSearch.overview.pageTitle', {
defaultMessage: 'Enterprise Search Overview',
});

useEffect(() => {
if (cluster) {
generateBreadcrumbs(cluster.cluster_name, { inEnterpriseSearch: true });
}
}, [cluster, generateBreadcrumbs]);

const getPageData = useCallback(async () => {
const bounds = services.data?.query.timefilter.timefilter.getBounds();
const url = `../api/monitoring/v1/clusters/${clusterUuid}/enterprise_search`;

const response = await services.http?.fetch<any>(url, {
method: 'POST',
body: JSON.stringify({
ccs,
timeRange: {
min: bounds.min.toISOString(),
max: bounds.max.toISOString(),
},
}),
});

setData(response);
}, [ccs, clusterUuid, services.data?.query.timefilter.timefilter, services.http]);

return (
<EntSearchTemplate
title={title}
pageTitle={pageTitle}
getPageData={getPageData}
data-test-subj="entSearchOverviewPage"
>
<div data-test-subj="entSearchOverviewPage">
{data && <EnterpriseSearchOverview {...data} onBrush={onBrush} zoomInfo={zoomInfo} />}
</div>
</EntSearchTemplate>
);
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import React from 'react';
import { get } from 'lodash';
import { formatNumber } from '../../../lib/format_number';
import {
BytesPercentageUsage,
ClusterItemContainer,
DisabledIfNoDataAndInSetupModeLink,
} from './helpers';
import { FormattedMessage } from '@kbn/i18n-react';
import { i18n } from '@kbn/i18n';
import {
EuiFlexGrid,
EuiFlexItem,
EuiTitle,
EuiPanel,
EuiDescriptionList,
EuiDescriptionListTitle,
EuiDescriptionListDescription,
EuiHorizontalRule,
EuiFlexGroup,
} from '@elastic/eui';
import { getSafeForExternalLink } from '../../../lib/get_safe_for_external_link';

export function EnterpriseSearchPanel(props) {
const { setupMode } = props;
const setupModeData = get(setupMode.data, 'enterprise_search');

return (
<ClusterItemContainer
{...props}
url="enterprise_search"
title={i18n.translate('xpack.monitoring.cluster.overview.entSearchPanel.entSearchTitle', {
defaultMessage: 'Enterprise Search',
})}
>
<EuiFlexGrid columns={4}>
<EuiFlexItem>
<EuiPanel paddingSize="m">
<EuiTitle size="s">
<h3>
<DisabledIfNoDataAndInSetupModeLink
setupModeEnabled={setupMode.enabled}
setupModeData={setupModeData}
href={getSafeForExternalLink('#/enterprise_search')}
aria-label={i18n.translate(
'xpack.monitoring.cluster.overview.entSearchPanel.overviewLinkAriaLabel',
{
defaultMessage: 'Enterprise Search Overview',
}
)}
data-test-subj="entSearchOverview"
>
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.overviewLinkLabel"
defaultMessage="Overview"
/>
</DisabledIfNoDataAndInSetupModeLink>
</h3>
</EuiTitle>
<EuiHorizontalRule margin="m" />
<EuiDescriptionList type="column">
<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.versionLabel"
defaultMessage="Version"
/>
</EuiDescriptionListTitle>

<EuiDescriptionListDescription data-test-subj="entSearchVersion">
{props.stats.versions[0] ||
i18n.translate(
'xpack.monitoring.cluster.overview.entSearchPanel.versionNotAvailableDescription',
{
defaultMessage: 'N/A',
}
)}
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.appSearchEngines"
defaultMessage="Engines"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="appSearchEngines">
{props.stats.appSearchEngines}
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.workplaceSearchOrgSources"
defaultMessage="Org Sources"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="workplaceSearchOrgSources">
{props.stats.workplaceSearchOrgSources}
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.workplaceSearchPrivateSources"
defaultMessage="Private Sources"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="workplaceSearchPrivateSources">
{props.stats.workplaceSearchPrivateSources}
</EuiDescriptionListDescription>
</EuiDescriptionList>
</EuiPanel>
</EuiFlexItem>

<EuiFlexItem>
<EuiPanel paddingSize="m">
<EuiFlexGroup justifyContent="spaceBetween">
<EuiFlexItem grow={false}>
<EuiTitle size="s">
<h3 data-test-subj="entSearchTotalNodes">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.nodesTotalLinkLabel"
defaultMessage="Nodes: {nodesTotal}"
values={{
nodesTotal: formatNumber(props.stats.totalInstances, 'int_commas'),
}}
/>
</h3>
</EuiTitle>
</EuiFlexItem>
</EuiFlexGroup>
<EuiHorizontalRule margin="m" />
<EuiDescriptionList type="column">
<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.memoryUsageLabel"
defaultMessage="Memory Usage"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="entSearchMemoryUsage">
<BytesPercentageUsage
usedBytes={props.stats.memUsed}
maxBytes={props.stats.memTotal}
/>
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.uptimeLabel"
defaultMessage="Uptime"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="entSearchUptime">
{formatNumber(props.stats.uptime, 'time_since')}
</EuiDescriptionListDescription>
</EuiDescriptionList>
</EuiPanel>
</EuiFlexItem>
</EuiFlexGrid>
</ClusterItemContainer>
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ export function ClusterItemContainer(props) {
logstash: 'logoLogstash',
beats: 'logoBeats',
apm: 'apmApp',
enterprise_search: 'logoEnterpriseSearch',
};
const icon = iconMap[props.url];

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { LogstashPanel } from './logstash_panel';
import { BeatsPanel } from './beats_panel';
import { EuiPage, EuiPageBody, EuiScreenReaderOnly } from '@elastic/eui';
import { ApmPanel } from './apm_panel';
import { EnterpriseSearchPanel } from './enterprise_search_panel';
import { FormattedMessage } from '@kbn/i18n-react';
import { STANDALONE_CLUSTER_CLUSTER_UUID } from '../../../../common/constants';

Expand Down Expand Up @@ -57,6 +58,12 @@ export function Overview(props) {
<BeatsPanel {...props.cluster.beats} setupMode={props.setupMode} alerts={props.alerts} />

<ApmPanel {...props.cluster.apm} setupMode={props.setupMode} alerts={props.alerts} />

<EnterpriseSearchPanel
{...props.cluster.enterpriseSearch}
setupMode={props.setupMode}
alerts={props.alerts}
/>
</EuiPageBody>
</EuiPage>
);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export { EnterpriseSearchOverview } from './overview';
Loading

0 comments on commit 7929123

Please sign in to comment.