Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(slo): new slo architecture #172224

Merged
merged 52 commits into from
Dec 12, 2023
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
52 commits
Select commit Hold shift + click to select a range
7af9763
feat(slo): add reset api (#170473)
kdelemme Nov 16, 2023
a6c43d1
Update mappings
kdelemme Nov 28, 2023
35a47ff
Add event ingested in rollup pipeline
kdelemme Nov 28, 2023
0054826
Simplify rollup mapping
kdelemme Nov 28, 2023
e501670
Cleanup installation of summary
kdelemme Nov 28, 2023
0f0ba55
Create summary transform generators
kdelemme Nov 29, 2023
36dd2c1
rename to summary transform generator
kdelemme Nov 29, 2023
ddac8ac
create template for summary pipeline
kdelemme Nov 29, 2023
978af77
Install pipeline and summary transform in create slo
kdelemme Nov 29, 2023
c03fc00
Handle rollback steps in create slo
kdelemme Nov 29, 2023
518f332
simplify rollback
kdelemme Nov 29, 2023
59828f8
Remove type cast
kdelemme Nov 29, 2023
31aae41
Remove summary transform and ingest pipeline when deleting an slo
kdelemme Nov 29, 2023
0c1151b
Update manage service to start/stop summary transform as well as roll…
kdelemme Nov 29, 2023
bd1e9a2
Update reset service to remove summary transform as well
kdelemme Nov 29, 2023
301af21
Updating an SLO handles the summary transform and pipeline
kdelemme Nov 29, 2023
0816470
Handle no revision bump change
kdelemme Nov 29, 2023
1e8ff20
Use new index
kdelemme Nov 29, 2023
59c4616
Add objective into summary mapping
kdelemme Nov 30, 2023
1fccef9
Fix test initialization
kdelemme Nov 30, 2023
9dcc83b
update snapshot tests
kdelemme Nov 30, 2023
ab41c81
Update rollup transform generator tests
kdelemme Nov 30, 2023
ff2ccd5
Refactor create slo tests
kdelemme Nov 30, 2023
90f6862
Update delete test
kdelemme Nov 30, 2023
07d01c9
Update manage tests
kdelemme Nov 30, 2023
4e9e79d
update reset test
kdelemme Nov 30, 2023
2d4fcd2
fix resource installer test
kdelemme Nov 30, 2023
93dda8e
update test for update slo
kdelemme Nov 30, 2023
32ca0de
[CI] Auto-commit changed files from 'node scripts/check_mappings_upda…
kibanamachine Nov 30, 2023
6da956a
Use first button
kdelemme Dec 1, 2023
674c9a3
Fix ftr test
kdelemme Dec 1, 2023
a5a32af
Fix rollup pipeline date formats
kdelemme Dec 1, 2023
552f2a5
make summary documents space aware
kdelemme Dec 1, 2023
3965b9d
Add slo.groupings into mappings for future multi group by support
kdelemme Dec 4, 2023
0477c2b
Merge branch 'main' into slo-new-architecture
kdelemme Dec 4, 2023
2409786
Merge branch 'main' into slo-new-architecture
kdelemme Dec 4, 2023
19c42ab
Fix merge conflicts
kdelemme Dec 4, 2023
f5b4d5b
fix import
kdelemme Dec 5, 2023
58a55cc
Merge branch 'main' into slo-new-architecture
kdelemme Dec 5, 2023
872b2b4
Fix timeslice and calendar aligned summary transform aggs query
kdelemme Dec 5, 2023
7df1340
use getActiveSpace
kdelemme Dec 5, 2023
6d8b5f3
Remove useless object.assign
kdelemme Dec 5, 2023
a131262
fix copy perPage limit
kdelemme Dec 5, 2023
51cc327
Merge branch 'main' into slo-new-architecture
kdelemme Dec 5, 2023
8936128
add slo name into summary description
kdelemme Dec 6, 2023
a7951c3
Merge branch 'main' into slo-new-architecture
kdelemme Dec 6, 2023
cf89bca
Add instanceId text mapping
kdelemme Dec 6, 2023
12e9746
Address comments
kdelemme Dec 8, 2023
ce00951
Merge branch 'main' into slo-new-architecture
kdelemme Dec 9, 2023
6ccd372
fix typo
kdelemme Dec 11, 2023
5319c6a
Merge branch 'main' into slo-new-architecture
kdelemme Dec 11, 2023
193e0f6
Merge branch 'main' into slo-new-architecture
kdelemme Dec 11, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -721,7 +721,8 @@
"indicator.params",
"indicator.type",
"name",
"tags"
"tags",
"version"
],
"threshold-explorer-view": [],
"observability-onboarding-state": [
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -2379,6 +2379,9 @@
},
"tags": {
"type": "keyword"
},
"version": {
"type": "long"
}
}
},
Expand Down
Original file line number Diff line number Diff line change
Expand Up @@ -142,7 +142,7 @@ describe('checking migration metadata changes on all registered SO types', () =>
"siem-ui-timeline": "d3de8ff3617be8f2a799d66b1471b9be6124bf40",
"siem-ui-timeline-note": "0a32fb776907f596bedca292b8c646496ae9c57b",
"siem-ui-timeline-pinned-event": "082daa3ce647b33873f6abccf340bdfa32057c8d",
"slo": "2048ab6791df2e1ae0936f29c20765cb8d2fcfaa",
"slo": "9a9995e4572de1839651c43b5fc4dc8276bb5815",
"space": "8de4ec513e9bbc6b2f1d635161d850be7747d38e",
"spaces-usage-stats": "3abca98713c52af8b30300e386c7779b3025a20e",
"synthetics-monitor": "33ddc4b8979f378edf58bcc7ba13c5c5b572f42d",
Expand Down
1 change: 1 addition & 0 deletions x-pack/packages/kbn-slo-schema/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -8,3 +8,4 @@
export * from './src/schema';
export * from './src/rest_specs';
export * from './src/models/duration';
export * from './src/models/pagination';
17 changes: 17 additions & 0 deletions x-pack/packages/kbn-slo-schema/src/models/pagination.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,17 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0; you may not use this file except in compliance with the Elastic License
* 2.0.
*/
export interface Paginated<T> {
total: number;
page: number;
perPage: number;
results: T[];
}

export interface Pagination {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could reuse this in Paginated<T>

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we could tackle that in this #172617

page: number;
perPage: number;
}
50 changes: 30 additions & 20 deletions x-pack/packages/kbn-slo-schema/src/rest_specs/slo.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
*/

import * as t from 'io-ts';
import { toBooleanRt } from '@kbn/io-ts-utils';
import {
allOrAnyString,
apmTransactionDurationIndicatorSchema,
Expand Down Expand Up @@ -109,6 +110,7 @@ const sloResponseSchema = t.intersection([
groupBy: allOrAnyString,
createdAt: dateType,
updatedAt: dateType,
version: t.number,
}),
t.partial({
instanceId: allOrAnyString,
Expand Down Expand Up @@ -157,6 +159,12 @@ const manageSLOParamsSchema = t.type({
path: t.type({ id: sloIdSchema }),
});

const resetSLOParamsSchema = t.type({
path: t.type({ id: sloIdSchema }),
});

const resetSLOResponseSchema = sloResponseSchema;

const updateSLOResponseSchema = sloResponseSchema;

const findSLOResponseSchema = t.type({
Expand All @@ -182,23 +190,21 @@ const fetchHistoricalSummaryResponseSchema = t.array(
})
);

/**
* The query params schema for /internal/observability/slo/_definitions
*
* @private
*/
const findSloDefinitionsParamsSchema = t.type({
query: t.type({
const findSloDefinitionsParamsSchema = t.partial({
query: t.partial({
search: t.string,
includeOutdatedOnly: toBooleanRt,
page: t.string,
perPage: t.string,
}),
});

/**
* The response schema for /internal/observability/slo/_definitions
*
* @private
*/
const findSloDefinitionsResponseSchema = t.array(sloResponseSchema);
const findSloDefinitionsResponseSchema = t.type({
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Could you reuse this schema to generate the Pagination type?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

More generally, I want to go through all these files and refactor them. I think the project grew, and we lost control on the schema/rest_specs/models. I think it would be good to spend some time thinking how to structure all these types.
Maybe having smaller building blocks, that can be reused to build the rest_specs and the models.
I think it's worth opening an issue for this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

page: t.number,
perPage: t.number,
total: t.number,
results: t.array(sloResponseSchema),
});

const getSLOBurnRatesResponseSchema = t.type({
burnRates: t.array(
Expand Down Expand Up @@ -244,6 +250,9 @@ type GetSLOResponse = t.OutputOf<typeof getSLOResponseSchema>;

type ManageSLOParams = t.TypeOf<typeof manageSLOParamsSchema.props.path>;

type ResetSLOParams = t.TypeOf<typeof resetSLOParamsSchema.props.path>;
type ResetSLOResponse = t.OutputOf<typeof resetSLOResponseSchema>;

type UpdateSLOInput = t.OutputOf<typeof updateSLOParamsSchema.props.body>;
type UpdateSLOParams = t.TypeOf<typeof updateSLOParamsSchema.props.body>;
type UpdateSLOResponse = t.OutputOf<typeof updateSLOResponseSchema>;
Expand All @@ -258,12 +267,8 @@ type FetchHistoricalSummaryParams = t.TypeOf<typeof fetchHistoricalSummaryParams
type FetchHistoricalSummaryResponse = t.OutputOf<typeof fetchHistoricalSummaryResponseSchema>;
type HistoricalSummaryResponse = t.OutputOf<typeof historicalSummarySchema>;

/**
* The response type for /internal/observability/slo/_definitions
*
* @private
*/
type FindSloDefinitionsResponse = t.OutputOf<typeof findSloDefinitionsResponseSchema>;
type FindSLODefinitionsParams = t.TypeOf<typeof findSloDefinitionsParamsSchema.props.query>;
type FindSLODefinitionsResponse = t.OutputOf<typeof findSloDefinitionsResponseSchema>;

type GetPreviewDataParams = t.TypeOf<typeof getPreviewDataParamsSchema.props.body>;
type GetPreviewDataResponse = t.OutputOf<typeof getPreviewDataResponseSchema>;
Expand Down Expand Up @@ -300,6 +305,8 @@ export {
findSloDefinitionsParamsSchema,
findSloDefinitionsResponseSchema,
manageSLOParamsSchema,
resetSLOParamsSchema,
resetSLOResponseSchema,
sloResponseSchema,
sloWithSummaryResponseSchema,
updateSLOParamsSchema,
Expand All @@ -325,8 +332,11 @@ export type {
FetchHistoricalSummaryParams,
FetchHistoricalSummaryResponse,
HistoricalSummaryResponse,
FindSloDefinitionsResponse,
FindSLODefinitionsParams,
FindSLODefinitionsResponse,
ManageSLOParams,
ResetSLOParams,
ResetSLOResponse,
SLOResponse,
SLOWithSummaryResponse,
UpdateSLOInput,
Expand Down
1 change: 1 addition & 0 deletions x-pack/packages/kbn-slo-schema/src/schema/slo.ts
Original file line number Diff line number Diff line change
Expand Up @@ -50,6 +50,7 @@ const sloSchema = t.type({
createdAt: dateType,
updatedAt: dateType,
groupBy: allOrAnyString,
version: t.number,
});

const sloWithSummarySchema = t.intersection([sloSchema, t.type({ summary: summarySchema })]);
Expand Down
3 changes: 2 additions & 1 deletion x-pack/packages/kbn-slo-schema/tsconfig.json
Original file line number Diff line number Diff line change
Expand Up @@ -11,7 +11,8 @@
"**/*.ts"
],
"kbn_references": [
"@kbn/std"
"@kbn/std",
"@kbn/io-ts-utils"
],
"exclude": [
"target/**/*",
Expand Down
4 changes: 2 additions & 2 deletions x-pack/performance/journeys/many_fields_transform.ts
Original file line number Diff line number Diff line change
Expand Up @@ -15,11 +15,11 @@ export const journey = new Journey({
.step('Go to Transforms', async ({ page, kbnUrl, kibanaPage }) => {
await page.goto(kbnUrl.get(`app/management/data/transform`));
await kibanaPage.waitForHeader();
await page.waitForSelector(subj('transformButtonCreate'));
await page.waitForSelector(subj('transformCreateFirstButton'));
await page.waitForSelector(subj('globalLoadingIndicator-hidden'));
})
.step('Go to data view selection', async ({ page }) => {
const createButtons = page.locator(subj('transformButtonCreate'));
const createButtons = page.locator(subj('transformCreateFirstButton'));
await createButtons.first().click();
await page.waitForSelector(subj('savedObjectsFinderTable'));
})
Expand Down
16 changes: 9 additions & 7 deletions x-pack/plugins/observability/common/slo/constants.ts
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@
* 2.0.
*/

export const SLO_RESOURCES_VERSION = 2;
export const SLO_SUMMARY_TRANSFORMS_VERSION = 3;
export const SLO_MODEL_VERSION = 2;
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

These values don't get bumped?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The SLO_MODEL_VERSION is bumped to 2 (set to 1 by the Saved Object migration, therefore existing SLO prior to the upgrade will have an internal version of 1). SLO_RESOURCES_VERSION is bumped to 3

export const SLO_RESOURCES_VERSION = 3;

export const SLO_COMPONENT_TEMPLATE_MAPPINGS_NAME = '.slo-observability.sli-mappings';
export const SLO_COMPONENT_TEMPLATE_SETTINGS_NAME = '.slo-observability.sli-settings';
Expand All @@ -17,21 +17,23 @@ export const SLO_INDEX_TEMPLATE_PATTERN = `.slo-observability.sli-*`;
export const SLO_DESTINATION_INDEX_NAME = `.slo-observability.sli-v${SLO_RESOURCES_VERSION}`;
export const SLO_DESTINATION_INDEX_PATTERN = `.slo-observability.sli-v${SLO_RESOURCES_VERSION}*`;

export const SLO_INGEST_PIPELINE_NAME = `.slo-observability.sli.pipeline`;
// slo-observability.sli-v<version>.(YYYY-MM-DD)
export const SLO_INGEST_PIPELINE_NAME = `.slo-observability.sli.pipeline-v${SLO_RESOURCES_VERSION}`;
export const SLO_INGEST_PIPELINE_INDEX_NAME_PREFIX = `.slo-observability.sli-v${SLO_RESOURCES_VERSION}.`;

export const SLO_SUMMARY_COMPONENT_TEMPLATE_MAPPINGS_NAME = '.slo-observability.summary-mappings';
export const SLO_SUMMARY_COMPONENT_TEMPLATE_SETTINGS_NAME = '.slo-observability.summary-settings';
export const SLO_SUMMARY_INDEX_TEMPLATE_NAME = '.slo-observability.summary';
export const SLO_SUMMARY_INDEX_TEMPLATE_PATTERN = `.slo-observability.summary-*`;

export const SLO_SUMMARY_TRANSFORM_NAME_PREFIX = 'slo-summary-';
export const SLO_SUMMARY_DESTINATION_INDEX_NAME = `.slo-observability.summary-v${SLO_RESOURCES_VERSION}`; // store the temporary summary document generated by transform
export const SLO_SUMMARY_TEMP_INDEX_NAME = `.slo-observability.summary-v${SLO_RESOURCES_VERSION}.temp`; // store the temporary summary document
export const SLO_SUMMARY_DESTINATION_INDEX_PATTERN = `.slo-observability.summary-v${SLO_RESOURCES_VERSION}*`; // include temp and non-temp summary indices

export const SLO_SUMMARY_INGEST_PIPELINE_NAME = `.slo-observability.summary.pipeline`;

export const getSLOTransformId = (sloId: string, sloRevision: number) =>
`slo-${sloId}-${sloRevision}`;

export const getSLOSummaryTransformId = (sloId: string, sloRevision: number) =>
`slo-summary-${sloId}-${sloRevision}`;

export const getSLOSummaryPipelineId = (sloId: string, sloRevision: number) =>
`.slo-observability.summary.pipeline-${sloId}-${sloRevision}`;
Loading
Loading