Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Enterprise Search Stack Monitoring #114303

Merged
merged 66 commits into from
Dec 7, 2021
Merged
Show file tree
Hide file tree
Changes from 62 commits
Commits
Show all changes
66 commits
Select commit Hold shift + click to select a range
79922f5
Added enterprise search panel, corrected queries
carlosdelest Aug 23, 2021
4ea8faf
Update the index pattern for Enterprise Search
kovyrin Oct 7, 2021
ccc7167
Typescript error ignore
kovyrin Oct 7, 2021
01f5a20
Our timestamp fields are called @timestamp (per ECS)
kovyrin Oct 7, 2021
5993a0b
Adjust Enterprise Search index patterns with the rest of monitoring p…
kovyrin Oct 7, 2021
fb64cac
Initial implementation of the Enterprise Search overview panel (healt…
kovyrin Oct 7, 2021
324c08b
Add a basic stub for enterprise search response fields
kovyrin Oct 7, 2021
c71d85b
Cleanup aggs configs
kovyrin Oct 7, 2021
d8c4900
Bring back a file deleted by mistake
kovyrin Oct 7, 2021
4a320f0
Started working on the overview page
kovyrin Oct 7, 2021
e04a604
Correctly use heap_max as the total heap
kovyrin Oct 7, 2021
785b816
Ent search breadcrumbs
kovyrin Oct 7, 2021
8107643
Simple overview
kovyrin Oct 7, 2021
a6427bc
Allow the cluster_uuid filter to be skipped while fetching metrics
kovyrin Oct 8, 2021
fd9b065
Cleanup
kovyrin Oct 8, 2021
174ffc1
Switch to module-level uuid field and use both types of events
kovyrin Oct 12, 2021
d8fb127
Add stats-based product usage metrics + apply filter paths to reduce …
kovyrin Oct 12, 2021
e73ec50
Change the name of the ent search overview class
kovyrin Oct 12, 2021
da39948
Move the standalone cluster hack in the the internal function
kovyrin Oct 12, 2021
aa3cc95
Change the overview page to show product usage metrics + introduce en…
kovyrin Oct 12, 2021
7739cd5
Cluster UUID is at the module level now
kovyrin Oct 12, 2021
6d7d95e
Simplify ent search pages structure, only have one overview page
kovyrin Oct 12, 2021
5507957
Fix ent search icon
kovyrin Oct 12, 2021
c9b4e24
Add total instances
kovyrin Oct 12, 2021
548a2a5
Product usage metric graphs
kovyrin Oct 12, 2021
4bd67ab
Simplify metrics loading in the overview page since we load all metri…
kovyrin Oct 12, 2021
a090c45
Add more enterprise search overview metrics
kovyrin Oct 12, 2021
36b7bb1
Avoid duplicate labels
kovyrin Oct 12, 2021
291f752
linting
kovyrin Oct 13, 2021
59ae44d
Revert "Simplify metrics loading in the overview page since we load a…
kovyrin Oct 13, 2021
8237244
Switch to multiple timeseries per graph
kovyrin Oct 13, 2021
9c75819
Reorder graphs and metrics for better experience
kovyrin Oct 13, 2021
9eab349
Merge branch 'master' into kovyrin/ent-search-monitoring
kovyrin Oct 13, 2021
b8ce664
Typescript fixes
kovyrin Oct 13, 2021
6f549a3
i18n fixes
kovyrin Oct 13, 2021
a9ceec8
Added a couple more JVM metrics
kovyrin Oct 13, 2021
71add11
Completely covered JVM metrics
kovyrin Oct 13, 2021
5d53e17
Convert Enterprise Search component to Typescript
kovyrin Oct 14, 2021
4dad950
Switch config setting back
kovyrin Oct 14, 2021
92cef3f
Remove the nodes link since it raises more questions than it solves
kovyrin Oct 14, 2021
7380fc4
Update jest snapshots with the new metrics
kovyrin Oct 14, 2021
65373e3
Merge branch 'master' into kovyrin/ent-search-monitoring
kibanamachine Oct 14, 2021
58e6943
Remove console statement
kovyrin Oct 15, 2021
01bbf69
Merge branch 'master' into kovyrin/ent-search-monitoring
kibanamachine Oct 15, 2021
213d956
Properly handle cases when aggregations return no data for Enterprise…
kovyrin Oct 15, 2021
960ab9b
Add a functional test for the Enterprise search cluster list panel
kovyrin Oct 15, 2021
497078e
Add a functional test for Enterprise Search overview page
kovyrin Oct 15, 2021
80086d4
Update multicluster API response fixture with the new enterprise sear…
kovyrin Oct 15, 2021
d08cac5
Default uptime value is 0
kovyrin Oct 15, 2021
aa7cebb
update overview fixture
kovyrin Oct 15, 2021
ae9c8d2
More fixture updates
kovyrin Oct 16, 2021
7450138
Merge branch 'master' into kovyrin/ent-search-monitoring
kibanamachine Oct 16, 2021
8d347aa
Merge branch 'master' into kovyrin/ent-search-monitoring
kibanamachine Oct 18, 2021
cdd06aa
Merge branch 'master' into kovyrin/ent-search-monitoring
kovyrin Oct 19, 2021
bfab7ac
Merge branch 'main' into kovyrin/ent-search-monitoring
kovyrin Nov 3, 2021
dfcfc7a
Merge branch 'main' into kovyrin/ent-search-monitoring
kibanamachine Nov 4, 2021
62180c6
Merge branch 'main' into kovyrin/ent-search-monitoring
kibanamachine Nov 10, 2021
e1c046a
Merge branch 'master' into kovyrin/ent-search-monitoring
kovyrin Dec 1, 2021
37fa2ff
Remove fixmes
kovyrin Dec 1, 2021
3a4951e
Fix imports
kovyrin Dec 1, 2021
68f3336
Properly export type
kovyrin Dec 1, 2021
5244518
Maybe fix the type checking error
kovyrin Dec 1, 2021
3d48164
PR Feedback
JasonStoltz Dec 2, 2021
3dd0643
TS fixes
JasonStoltz Dec 3, 2021
26e5e8b
Merge branch 'main' into kovyrin/ent-search-monitoring
kibanamachine Dec 3, 2021
3b156b8
Merge branch 'main' into kovyrin/ent-search-monitoring
kibanamachine Dec 7, 2021
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
8 changes: 8 additions & 0 deletions x-pack/plugins/monitoring/common/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -129,6 +129,7 @@ export const INDEX_PATTERN_LOGSTASH = '.monitoring-logstash-6-*,.monitoring-logs
export const INDEX_PATTERN_BEATS = '.monitoring-beats-6-*,.monitoring-beats-7-*';
export const INDEX_ALERTS = '.monitoring-alerts-6*,.monitoring-alerts-7*';
export const INDEX_PATTERN_ELASTICSEARCH = '.monitoring-es-6-*,.monitoring-es-7-*';
export const INDEX_PATTERN_ENTERPRISE_SEARCH = '.monitoring-ent-search-*';

// This is the unique token that exists in monitoring indices collected by metricbeat
export const METRICBEAT_INDEX_NAME_UNIQUE_TOKEN = '-mb-';
Expand Down Expand Up @@ -158,6 +159,7 @@ export const CODE_PATH_LOGSTASH = 'logstash';
export const CODE_PATH_APM = 'apm';
export const CODE_PATH_LICENSE = 'license';
export const CODE_PATH_LOGS = 'logs';
export const CODE_PATH_ENTERPRISE_SEARCH = 'enterprise_search';

/**
* The header sent by telemetry service when hitting Elasticsearch to identify query source
Expand All @@ -177,6 +179,12 @@ export const KIBANA_SYSTEM_ID = 'kibana';
*/
export const BEATS_SYSTEM_ID = 'beats';

/**
* The name of the Enterprise Search System ID used to publish and look up Enterprise Search stats through the Monitoring system.
* @type {string}
*/
export const ENTERPRISE_SEARCH_SYSTEM_ID = 'enterprise_search';

/**
* The name of the Apm System ID used to publish and look up Apm stats through the Monitoring system.
* @type {string}
Expand Down
3 changes: 3 additions & 0 deletions x-pack/plugins/monitoring/common/types/es.ts
Original file line number Diff line number Diff line change
Expand Up @@ -677,6 +677,9 @@ export interface ElasticsearchMetricbeatSource {
};
};
};
enterprisesearch?: {
cluster_uuid?: string;
};
}

export type ElasticsearchSource = ElasticsearchLegacySource & ElasticsearchMetricbeatSource;
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -186,6 +186,16 @@ function getApmBreadcrumbs(mainInstance: any) {
return breadcrumbs;
}

// generate Enterprise Search breadcrumbs
function getEnterpriseSearchBreadcrumbs(mainInstance: any) {
const entSearchLabel = i18n.translate('xpack.monitoring.breadcrumbs.entSearchLabel', {
defaultMessage: 'Enterprise Search',
});
const breadcrumbs = [];
breadcrumbs.push(createCrumb('#/enterprise_search', entSearchLabel));
return breadcrumbs;
}

function buildBreadcrumbs(clusterName: string, mainInstance?: any | null) {
const homeCrumb = i18n.translate('xpack.monitoring.breadcrumbs.clustersLabel', {
defaultMessage: 'Clusters',
Expand All @@ -212,6 +222,9 @@ function buildBreadcrumbs(clusterName: string, mainInstance?: any | null) {
if (mainInstance?.inApm) {
breadcrumbs = breadcrumbs.concat(getApmBreadcrumbs(mainInstance));
}
if (mainInstance?.inEnterpriseSearch) {
breadcrumbs = breadcrumbs.concat(getEnterpriseSearchBreadcrumbs(mainInstance));
}

return breadcrumbs;
}
Expand Down
9 changes: 9 additions & 0 deletions x-pack/plugins/monitoring/public/application/index.tsx
Original file line number Diff line number Diff line change
Expand Up @@ -28,6 +28,7 @@ import {
CODE_PATH_KIBANA,
CODE_PATH_LOGSTASH,
CODE_PATH_APM,
CODE_PATH_ENTERPRISE_SEARCH,
} from '../../common/constants';
import { BeatsInstancePage } from './pages/beats/instance';
import { ApmOverviewPage, ApmInstancesPage, ApmInstancePage } from './pages/apm';
Expand All @@ -43,6 +44,7 @@ import { ElasticsearchMLJobsPage } from './pages/elasticsearch/ml_jobs_page';
import { ElasticsearchNodeAdvancedPage } from './pages/elasticsearch/node_advanced_page';
import { ElasticsearchCcrPage } from './pages/elasticsearch/ccr_page';
import { ElasticsearchCcrShardPage } from './pages/elasticsearch/ccr_shard_page';
import { EntSearchOverviewPage } from './pages/enterprise_search/overview';
import { MonitoringTimeContainer } from './hooks/use_monitoring_time';
import { BreadcrumbContainer } from './hooks/use_breadcrumbs';
import { HeaderActionMenuContext } from './contexts/header_action_menu_context';
Expand Down Expand Up @@ -312,6 +314,13 @@ const MonitoringApp: React.FC<{
fetchAllClusters={false}
/>

<RouteInit
path="/enterprise_search"
component={EntSearchOverviewPage}
codePaths={[CODE_PATH_ENTERPRISE_SEARCH]}
fetchAllClusters={false}
/>

<Redirect
to={{
pathname: '/loading',
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,15 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
import React from 'react';
import { PageTemplate } from '../page_template';
import { PageTemplateProps } from '../page_template';

type EntSearchTemplateProps = PageTemplateProps;

export const EntSearchTemplate: React.FC<EntSearchTemplateProps> = (props) => {
return <PageTemplate {...props} product="enterprise_search" />;
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import React, { useContext, useState, useCallback, useEffect } from 'react';
import { i18n } from '@kbn/i18n';
import { find } from 'lodash';
import { ComponentProps } from '../../route_init';
import { EntSearchTemplate } from './ent_search_template';
import { GlobalStateContext } from '../../contexts/global_state_context';
import { useCharts } from '../../hooks/use_charts';
import { useKibana } from '../../../../../../../src/plugins/kibana_react/public';
import { EnterpriseSearchOverview } from '../../../components/enterprise_search/overview';
import { BreadcrumbContainer } from '../../hooks/use_breadcrumbs';

export const EntSearchOverviewPage: React.FC<ComponentProps> = ({ clusters }) => {
const globalState = useContext(GlobalStateContext);
const { zoomInfo, onBrush } = useCharts();
const { services } = useKibana<{ data: any }>();
const clusterUuid = globalState.cluster_uuid;
const ccs = globalState.ccs;
const { generate: generateBreadcrumbs } = useContext(BreadcrumbContainer.Context);
const cluster = find(clusters, {
cluster_uuid: clusterUuid,
}) as any;

const [data, setData] = useState(null);

const title = i18n.translate('xpack.monitoring.entSearch.overview.routeTitle', {
defaultMessage: 'Enterprise Search - Overview',
});

const pageTitle = i18n.translate('xpack.monitoring.entSearch.overview.pageTitle', {
defaultMessage: 'Enterprise Search Overview',
});

useEffect(() => {
if (cluster) {
generateBreadcrumbs(cluster.cluster_name, { inEnterpriseSearch: true });
}
}, [cluster, generateBreadcrumbs]);

const getPageData = useCallback(async () => {
const bounds = services.data?.query.timefilter.timefilter.getBounds();
const url = `../api/monitoring/v1/clusters/${clusterUuid}/enterprise_search`;

const response = await services.http?.fetch<any>(url, {
method: 'POST',
body: JSON.stringify({
ccs,
timeRange: {
min: bounds.min.toISOString(),
max: bounds.max.toISOString(),
},
}),
});

setData(response);
}, [ccs, clusterUuid, services.data?.query.timefilter.timefilter, services.http]);

const renderOverview = (overviewData: any) => {
phillipb marked this conversation as resolved.
Show resolved Hide resolved
if (overviewData === null) {
return null;
}
return <EnterpriseSearchOverview {...overviewData} onBrush={onBrush} zoomInfo={zoomInfo} />;
};

return (
<EntSearchTemplate
title={title}
pageTitle={pageTitle}
getPageData={getPageData}
data-test-subj="entSearchOverviewPage"
>
<div data-test-subj="entSearchOverviewPage">{renderOverview(data)}</div>
</EntSearchTemplate>
);
};
Original file line number Diff line number Diff line change
@@ -0,0 +1,166 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

import React from 'react';
import { get } from 'lodash';
import { formatNumber } from '../../../lib/format_number';
import {
BytesPercentageUsage,
ClusterItemContainer,
DisabledIfNoDataAndInSetupModeLink,
} from './helpers';
import { FormattedMessage } from '@kbn/i18n-react';
import { i18n } from '@kbn/i18n';
import {
EuiFlexGrid,
EuiFlexItem,
EuiTitle,
EuiPanel,
EuiDescriptionList,
EuiDescriptionListTitle,
EuiDescriptionListDescription,
EuiHorizontalRule,
EuiFlexGroup,
} from '@elastic/eui';
import { getSafeForExternalLink } from '../../../lib/get_safe_for_external_link';

export function EnterpriseSearchPanel(props) {
const { setupMode } = props;
const setupModeData = get(setupMode.data, 'enterprise_search');

return (
<ClusterItemContainer
{...props}
url="enterprise_search"
title={i18n.translate('xpack.monitoring.cluster.overview.entSearchPanel.entSearchTitle', {
defaultMessage: 'Enterprise Search',
})}
>
<EuiFlexGrid columns={4}>
<EuiFlexItem>
<EuiPanel paddingSize="m">
<EuiTitle size="s">
<h3>
<DisabledIfNoDataAndInSetupModeLink
setupModeEnabled={setupMode.enabled}
setupModeData={setupModeData}
href={getSafeForExternalLink('#/enterprise_search')}
aria-label={i18n.translate(
'xpack.monitoring.cluster.overview.entSearchPanel.overviewLinkAriaLabel',
{
defaultMessage: 'Enterprise Search Overview',
}
)}
data-test-subj="entSearchOverview"
>
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.overviewLinkLabel"
defaultMessage="Overview"
/>
</DisabledIfNoDataAndInSetupModeLink>
</h3>
</EuiTitle>
<EuiHorizontalRule margin="m" />
<EuiDescriptionList type="column">
<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.versionLabel"
defaultMessage="Version"
/>
</EuiDescriptionListTitle>

<EuiDescriptionListDescription data-test-subj="entSearchVersion">
{props.stats.versions[0] ||
i18n.translate(
'xpack.monitoring.cluster.overview.entSearchPanel.versionNotAvailableDescription',
{
defaultMessage: 'N/A',
}
)}
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.appSearchEngines"
defaultMessage="Engines"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="appSearchEngines">
{props.stats.appSearchEngines}
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.workplaceSearchOrgSources"
defaultMessage="Org Sources"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="workplaceSearchOrgSources">
{props.stats.workplaceSearchOrgSources}
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.workplaceSearchPrivateSources"
defaultMessage="Private Sources"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="workplaceSearchPrivateSources">
{props.stats.workplaceSearchPrivateSources}
</EuiDescriptionListDescription>
</EuiDescriptionList>
</EuiPanel>
</EuiFlexItem>

<EuiFlexItem>
<EuiPanel paddingSize="m">
<EuiFlexGroup justifyContent="spaceBetween">
<EuiFlexItem grow={false}>
<EuiTitle size="s">
<h3 data-test-subj="entSearchTotalNodes">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.nodesTotalLinkLabel"
defaultMessage="Nodes: {nodesTotal}"
values={{
nodesTotal: formatNumber(props.stats.totalInstances, 'int_commas'),
}}
/>
</h3>
</EuiTitle>
</EuiFlexItem>
</EuiFlexGroup>
<EuiHorizontalRule margin="m" />
<EuiDescriptionList type="column">
<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.memoryUsageLabel"
defaultMessage="Memory Usage"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="entSearchMemoryUsage">
<BytesPercentageUsage
usedBytes={props.stats.memUsed}
maxBytes={props.stats.memTotal}
/>
</EuiDescriptionListDescription>

<EuiDescriptionListTitle className="eui-textBreakWord">
<FormattedMessage
id="xpack.monitoring.cluster.overview.entSearchPanel.uptimeLabel"
defaultMessage="Uptime"
/>
</EuiDescriptionListTitle>
<EuiDescriptionListDescription data-test-subj="entSearchUptime">
{formatNumber(props.stats.uptime, 'time_since')}
</EuiDescriptionListDescription>
</EuiDescriptionList>
</EuiPanel>
</EuiFlexItem>
</EuiFlexGrid>
</ClusterItemContainer>
);
}
Original file line number Diff line number Diff line change
Expand Up @@ -94,6 +94,7 @@ export function ClusterItemContainer(props) {
logstash: 'logoLogstash',
beats: 'logoBeats',
apm: 'apmApp',
enterprise_search: 'logoEnterpriseSearch',
};
const icon = iconMap[props.url];

Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,7 @@ import { LogstashPanel } from './logstash_panel';
import { BeatsPanel } from './beats_panel';
import { EuiPage, EuiPageBody, EuiScreenReaderOnly } from '@elastic/eui';
import { ApmPanel } from './apm_panel';
import { EnterpriseSearchPanel } from './enterprise_search_panel';
import { FormattedMessage } from '@kbn/i18n-react';
import { STANDALONE_CLUSTER_CLUSTER_UUID } from '../../../../common/constants';

Expand Down Expand Up @@ -57,6 +58,12 @@ export function Overview(props) {
<BeatsPanel {...props.cluster.beats} setupMode={props.setupMode} alerts={props.alerts} />

<ApmPanel {...props.cluster.apm} setupMode={props.setupMode} alerts={props.alerts} />

<EnterpriseSearchPanel
{...props.cluster.enterpriseSearch}
setupMode={props.setupMode}
alerts={props.alerts}
/>
</EuiPageBody>
</EuiPage>
);
Expand Down
Original file line number Diff line number Diff line change
@@ -0,0 +1,8 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/

export { EnterpriseSearchOverview } from './overview';
Loading