Skip to content

Commit

Permalink
[Response Ops][Alerting] Install resources needed for framework alert…
Browse files Browse the repository at this point in the history
…s-as-data (#145581)

Resolves #145100

## Summary

* Adds alerting config for `enableFrameworkAlerts`
* When `xpack.alerting.enableFrameworkAlerts=true`, `AlertsService` is
initialized during the alerting plugin setup phase
* Adds optional `alerts` definition to the `RuleType` definition which
allows a rule type to specify a context and a field mapping for that
context
* `AlertsService.initialize()` 
* installs an ILM policy that will be used by all alerts-as-data indices
- `alerts-default-ilm-policy`
* installs a component template that will be used by all alerts-as-data
indices - `alerts-common-component-template`, including all the mappings
for fields needed by the framework
* Rule type registration - when a rule type with an `alerts` definition
is registered, the context and field map are registered with the
`AlertService`. When `AlertsService` initialization is completed
successfully, context specific resources are installed.
* Context specific resources:
* installs context specific component template that reflects the
registered field map - `alerts-${context}-component-template`
* installs context specific index template for the default namespace -
`.alerts-${context}-default-template`
* creates context specific concrete write index for the default
namespace - `.alerts-${context}-default-000001`
* Resource installation retries for transient ES errors and
short-circuits when a timeout value is reached

## Notes
* We explore the possibility of creating a single index template per
context and re-using it for space-aware concrete indices but a rollover
alias (which must be space-aware) must be attached to an index template
so it is not feasible to only have a single index template
* There will be a followup issue for create space-specific index
templates/indices as needed to support detection rules desire to
separate alerts by space.

## To Verify

* set `xpack.alerting.enableFrameworkAlerts: true` in your kibana.yml
* modify
`x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts` to
define a custom field mapping for the `stack` context

```
--- a/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
+++ b/x-pack/plugins/stack_alerts/server/rule_types/es_query/rule_type.ts
@@ -187,5 +187,14 @@ export function getRuleType(
     },
     producer: STACK_ALERTS_FEATURE_ID,
     doesSetRecoveryContext: true,
+    alerts: {
+      context: 'stack',
+      fieldMap: {
+        'kibana.alert.threshold': {
+          required: false,
+          type: 'long',
+        },
+      },
+    },
   };
 }

```
* start up ES and Kibana and verify that the correct resources are
installed. The following should be created:
  * ILM policy - `alerts-default-ilm-policy`
  * Common component template - `alerts-common-component-template`
  * Stack component template - `alerts-stack-component-template`
* Stack index template for default space -
`.alerts-stack-default-template`
  * Stack write index for default space - `.alerts-stack-default-000001`
* verify that the index template uses both component templates and that
the expected mappings are applied to the index
* try making non-destructive, additive only changes to the common
component template or the stack context field mapping and verify that
the concrete index mappings are updated.

### Checklist

- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
  • Loading branch information
ymao1 and kibanamachine authored Jan 24, 2023
1 parent 4c626f1 commit b7a0204
Show file tree
Hide file tree
Showing 30 changed files with 2,959 additions and 58 deletions.
1 change: 1 addition & 0 deletions packages/kbn-rule-data-utils/index.ts
Original file line number Diff line number Diff line change
Expand Up @@ -6,6 +6,7 @@
* Side Public License, v 1.
*/

export * from './src/default_alerts_as_data';
export * from './src/technical_field_names';
export * from './src/alerts_as_data_rbac';
export * from './src/alerts_as_data_severity';
Expand Down
141 changes: 141 additions & 0 deletions packages/kbn-rule-data-utils/src/default_alerts_as_data.ts
Original file line number Diff line number Diff line change
@@ -0,0 +1,141 @@
/*
* Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
* or more contributor license agreements. Licensed under the Elastic License
* 2.0 and the Server Side Public License, v 1; you may not use this file except
* in compliance with, at your election, the Elastic License 2.0 or the Server
* Side Public License, v 1.
*/

import { ValuesType } from 'utility-types';

const KIBANA_NAMESPACE = 'kibana' as const;
const ALERT_NAMESPACE = `${KIBANA_NAMESPACE}.alert` as const;
const ALERT_RULE_NAMESPACE = `${ALERT_NAMESPACE}.rule` as const;

// kibana.space_ids - space ID(s) of the rule that created this alert
const SPACE_IDS = `${KIBANA_NAMESPACE}.space_ids` as const;

// kibana.version - Kibana version that this alert was created
const VERSION = `${KIBANA_NAMESPACE}.version` as const;

// kibana.alert.action_group - framework action group ID for this alert
const ALERT_ACTION_GROUP = `${ALERT_NAMESPACE}.action_group` as const;

// kibana.alert.duration.us - alert duration in nanoseconds - updated each execution
// that the alert is active
const ALERT_DURATION = `${ALERT_NAMESPACE}.duration.us` as const;

// kibana.alert.end - timestamp when the alert is auto-recovered by the framework
const ALERT_END = `${ALERT_NAMESPACE}.end` as const;

// kibana.alert.flapping - whether the alert is currently in a flapping state
const ALERT_FLAPPING = `${ALERT_NAMESPACE}.flapping` as const;

// kibana.alert.id - alert ID, also known as alert instance ID
const ALERT_ID = `${ALERT_NAMESPACE}.id` as const;

// kibana.alert.reason - human readable reason that this alert is active
const ALERT_REASON = `${ALERT_NAMESPACE}.reason` as const;

// kibana.alert.start - timestamp when the alert is first active
const ALERT_START = `${ALERT_NAMESPACE}.start` as const;

// kibana.alert.status - active/recovered status of alert
const ALERT_STATUS = `${ALERT_NAMESPACE}.status` as const;

// kibana.alert.time_range - time range of alert from kibana.alert.start to now
const ALERT_TIME_RANGE = `${ALERT_NAMESPACE}.time_range` as const;

// kibana.alert.uuid - unique ID for the active span of this alert
const ALERT_UUID = `${ALERT_NAMESPACE}.uuid` as const;

// kibana.alert.workflow_status - open/closed status of alert
const ALERT_WORKFLOW_STATUS = `${ALERT_NAMESPACE}.workflow_status` as const;

// kibana.alert.rule.category - rule type name for rule that generated this alert
const ALERT_RULE_CATEGORY = `${ALERT_RULE_NAMESPACE}.category` as const;

// kibana.alert.rule.consumer - consumer for rule that generated this alert
const ALERT_RULE_CONSUMER = `${ALERT_RULE_NAMESPACE}.consumer` as const;

// kibana.alert.rule.execution.uuid - unique ID for the rule execution that generated this alert
const ALERT_RULE_EXECUTION_UUID = `${ALERT_RULE_NAMESPACE}.execution.uuid` as const;

// kibana.alert.rule.name - rule name for rule that generated this alert
const ALERT_RULE_NAME = `${ALERT_RULE_NAMESPACE}.name` as const;

// kibana.alert.rule.parameters - rule parameters for rule that generated this alert
const ALERT_RULE_PARAMETERS = `${ALERT_RULE_NAMESPACE}.parameters` as const;

// kibana.alert.rule.producer - rule type producer for rule that generated this alert
const ALERT_RULE_PRODUCER = `${ALERT_RULE_NAMESPACE}.producer` as const;

// kibana.alert.rule.tags - rule tags for rule that generated this alert
const ALERT_RULE_TAGS = `${ALERT_RULE_NAMESPACE}.tags` as const;

// kibana.alert.rule_type_id - rule type id for rule that generated this alert
const ALERT_RULE_TYPE_ID = `${ALERT_RULE_NAMESPACE}.rule_type_id` as const;

// kibana.alert.rule.uuid - rule ID for rule that generated this alert
const ALERT_RULE_UUID = `${ALERT_RULE_NAMESPACE}.uuid` as const;

const namespaces = {
KIBANA_NAMESPACE,
ALERT_NAMESPACE,
ALERT_RULE_NAMESPACE,
};

const fields = {
ALERT_ACTION_GROUP,
ALERT_DURATION,
ALERT_END,
ALERT_FLAPPING,
ALERT_ID,
ALERT_REASON,
ALERT_RULE_CATEGORY,
ALERT_RULE_CONSUMER,
ALERT_RULE_EXECUTION_UUID,
ALERT_RULE_NAME,
ALERT_RULE_PARAMETERS,
ALERT_RULE_PRODUCER,
ALERT_RULE_TAGS,
ALERT_RULE_TYPE_ID,
ALERT_RULE_UUID,
ALERT_START,
ALERT_STATUS,
ALERT_TIME_RANGE,
ALERT_UUID,
ALERT_WORKFLOW_STATUS,
SPACE_IDS,
VERSION,
};

export {
ALERT_ACTION_GROUP,
ALERT_DURATION,
ALERT_END,
ALERT_FLAPPING,
ALERT_ID,
ALERT_REASON,
ALERT_RULE_CATEGORY,
ALERT_RULE_CONSUMER,
ALERT_RULE_EXECUTION_UUID,
ALERT_RULE_NAME,
ALERT_RULE_PARAMETERS,
ALERT_RULE_PRODUCER,
ALERT_RULE_TAGS,
ALERT_RULE_TYPE_ID,
ALERT_RULE_UUID,
ALERT_START,
ALERT_STATUS,
ALERT_TIME_RANGE,
ALERT_UUID,
ALERT_WORKFLOW_STATUS,
SPACE_IDS,
VERSION,
ALERT_NAMESPACE,
ALERT_RULE_NAMESPACE,
KIBANA_NAMESPACE,
};

export type DefaultAlertFieldName = ValuesType<typeof fields & typeof namespaces>;
82 changes: 26 additions & 56 deletions packages/kbn-rule-data-utils/src/technical_field_names.ts
Original file line number Diff line number Diff line change
Expand Up @@ -7,41 +7,51 @@
*/

import { ValuesType } from 'utility-types';
import {
KIBANA_NAMESPACE,
ALERT_ACTION_GROUP,
ALERT_DURATION,
ALERT_END,
ALERT_FLAPPING,
ALERT_REASON,
ALERT_RULE_CATEGORY,
ALERT_RULE_CONSUMER,
ALERT_RULE_EXECUTION_UUID,
ALERT_RULE_NAME,
ALERT_RULE_PARAMETERS,
ALERT_RULE_PRODUCER,
ALERT_RULE_TAGS,
ALERT_RULE_TYPE_ID,
ALERT_RULE_UUID,
ALERT_START,
ALERT_STATUS,
ALERT_TIME_RANGE,
ALERT_UUID,
ALERT_WORKFLOW_STATUS,
SPACE_IDS,
VERSION,
ALERT_NAMESPACE,
ALERT_RULE_NAMESPACE,
} from './default_alerts_as_data';

const KIBANA_NAMESPACE = 'kibana' as const;

const ALERT_NAMESPACE = `${KIBANA_NAMESPACE}.alert` as const;
const ALERT_RULE_NAMESPACE = `${ALERT_NAMESPACE}.rule` as const;
const ALERT_RULE_THREAT_NAMESPACE = `${ALERT_RULE_NAMESPACE}.threat` as const;

const ECS_VERSION = 'ecs.version' as const;
const EVENT_ACTION = 'event.action' as const;
const EVENT_KIND = 'event.kind' as const;
const EVENT_MODULE = 'event.module' as const;
const SPACE_IDS = `${KIBANA_NAMESPACE}.space_ids` as const;
const TAGS = 'tags' as const;
const TIMESTAMP = '@timestamp' as const;
const VERSION = `${KIBANA_NAMESPACE}.version` as const;

// Fields pertaining to the alert
const ALERT_ACTION_GROUP = `${ALERT_NAMESPACE}.action_group` as const;
const ALERT_BUILDING_BLOCK_TYPE = `${ALERT_NAMESPACE}.building_block_type` as const;
const ALERT_DURATION = `${ALERT_NAMESPACE}.duration.us` as const;
const ALERT_END = `${ALERT_NAMESPACE}.end` as const;
const ALERT_EVALUATION_THRESHOLD = `${ALERT_NAMESPACE}.evaluation.threshold` as const;
const ALERT_EVALUATION_VALUE = `${ALERT_NAMESPACE}.evaluation.value` as const;
const ALERT_FLAPPING = `${ALERT_NAMESPACE}.flapping` as const;
const ALERT_INSTANCE_ID = `${ALERT_NAMESPACE}.instance.id` as const;
const ALERT_REASON = `${ALERT_NAMESPACE}.reason` as const;
const ALERT_RISK_SCORE = `${ALERT_NAMESPACE}.risk_score` as const;
const ALERT_SEVERITY = `${ALERT_NAMESPACE}.severity` as const;
const ALERT_START = `${ALERT_NAMESPACE}.start` as const;
const ALERT_TIME_RANGE = `${ALERT_NAMESPACE}.time_range` as const;
const ALERT_STATUS = `${ALERT_NAMESPACE}.status` as const;
const ALERT_SYSTEM_STATUS = `${ALERT_NAMESPACE}.system_status` as const;
const ALERT_UUID = `${ALERT_NAMESPACE}.uuid` as const;
const ALERT_WORKFLOW_REASON = `${ALERT_NAMESPACE}.workflow_reason` as const;
const ALERT_WORKFLOW_STATUS = `${ALERT_NAMESPACE}.workflow_status` as const;
const ALERT_WORKFLOW_USER = `${ALERT_NAMESPACE}.workflow_user` as const;
const ALERT_SUPPRESSION_META = `${ALERT_NAMESPACE}.suppression` as const;
const ALERT_SUPPRESSION_TERMS = `${ALERT_SUPPRESSION_META}.terms` as const;
Expand All @@ -61,22 +71,16 @@ const ALERT_RULE_CREATED_BY = `${ALERT_RULE_NAMESPACE}.created_by` as const;
const ALERT_RULE_DESCRIPTION = `${ALERT_RULE_NAMESPACE}.description` as const;
const ALERT_RULE_ENABLED = `${ALERT_RULE_NAMESPACE}.enabled` as const;
const ALERT_RULE_EXCEPTIONS_LIST = `${ALERT_RULE_NAMESPACE}.exceptions_list` as const;
const ALERT_RULE_EXECUTION_UUID = `${ALERT_RULE_NAMESPACE}.execution.uuid` as const;
const ALERT_RULE_FROM = `${ALERT_RULE_NAMESPACE}.from` as const;
const ALERT_RULE_INTERVAL = `${ALERT_RULE_NAMESPACE}.interval` as const;
const ALERT_RULE_LICENSE = `${ALERT_RULE_NAMESPACE}.license` as const;
const ALERT_RULE_CATEGORY = `${ALERT_RULE_NAMESPACE}.category` as const;
const ALERT_RULE_NAME = `${ALERT_RULE_NAMESPACE}.name` as const;
const ALERT_RULE_NAMESPACE_FIELD = `${ALERT_RULE_NAMESPACE}.namespace` as const;
const ALERT_RULE_NOTE = `${ALERT_RULE_NAMESPACE}.note` as const;
const ALERT_RULE_PARAMETERS = `${ALERT_RULE_NAMESPACE}.parameters` as const;
const ALERT_RULE_REFERENCES = `${ALERT_RULE_NAMESPACE}.references` as const;
const ALERT_RULE_RULE_ID = `${ALERT_RULE_NAMESPACE}.rule_id` as const;
const ALERT_RULE_RULE_NAME_OVERRIDE = `${ALERT_RULE_NAMESPACE}.rule_name_override` as const;
const ALERT_RULE_TAGS = `${ALERT_RULE_NAMESPACE}.tags` as const;
const ALERT_RULE_TO = `${ALERT_RULE_NAMESPACE}.to` as const;
const ALERT_RULE_TYPE = `${ALERT_RULE_NAMESPACE}.type` as const;
const ALERT_RULE_TYPE_ID = `${ALERT_RULE_NAMESPACE}.rule_type_id` as const;
const ALERT_RULE_UPDATED_AT = `${ALERT_RULE_NAMESPACE}.updated_at` as const;
const ALERT_RULE_UPDATED_BY = `${ALERT_RULE_NAMESPACE}.updated_by` as const;
const ALERT_RULE_VERSION = `${ALERT_RULE_NAMESPACE}.version` as const;
Expand All @@ -97,16 +101,6 @@ const ALERT_THREAT_TECHNIQUE_SUBTECHNIQUE_NAME =
const ALERT_THREAT_TECHNIQUE_SUBTECHNIQUE_REFERENCE =
`${ALERT_RULE_THREAT_NAMESPACE}.technique.subtechnique.reference` as const;

// the feature instantiating a rule type.
// Rule created in stack --> alerts
// Rule created in siem --> siem
const ALERT_RULE_CONSUMER = `${ALERT_RULE_NAMESPACE}.consumer` as const;
// the plugin that registered the rule type.
// Rule type apm.error_rate --> apm
// Rule type siem.signals --> siem
const ALERT_RULE_PRODUCER = `${ALERT_RULE_NAMESPACE}.producer` as const;
const ALERT_RULE_UUID = `${ALERT_RULE_NAMESPACE}.uuid` as const;

const namespaces = {
KIBANA_NAMESPACE,
ALERT_NAMESPACE,
Expand Down Expand Up @@ -189,23 +183,12 @@ const fields = {
};

export {
ALERT_ACTION_GROUP,
ALERT_BUILDING_BLOCK_TYPE,
ALERT_DURATION,
ALERT_END,
ALERT_EVALUATION_THRESHOLD,
ALERT_EVALUATION_VALUE,
ALERT_FLAPPING,
ALERT_INSTANCE_ID,
ALERT_NAMESPACE,
ALERT_RULE_NAMESPACE,
ALERT_RULE_CONSUMER,
ALERT_RULE_PRODUCER,
ALERT_REASON,
ALERT_RISK_SCORE,
ALERT_STATUS,
ALERT_WORKFLOW_REASON,
ALERT_WORKFLOW_STATUS,
ALERT_WORKFLOW_USER,
ALERT_CASE_IDS,
ALERT_RULE_AUTHOR,
Expand All @@ -214,36 +197,25 @@ export {
ALERT_RULE_DESCRIPTION,
ALERT_RULE_ENABLED,
ALERT_RULE_EXCEPTIONS_LIST,
ALERT_RULE_EXECUTION_UUID,
ALERT_RULE_FROM,
ALERT_RULE_INTERVAL,
ALERT_RULE_LICENSE,
ALERT_RULE_NAME,
ALERT_RULE_NAMESPACE_FIELD,
ALERT_RULE_NOTE,
ALERT_RULE_PARAMETERS,
ALERT_RULE_REFERENCES,
ALERT_RULE_RULE_ID,
ALERT_RULE_RULE_NAME_OVERRIDE,
ALERT_RULE_TAGS,
ALERT_RULE_TO,
ALERT_RULE_TYPE,
ALERT_RULE_TYPE_ID,
ALERT_RULE_UPDATED_AT,
ALERT_RULE_UPDATED_BY,
ALERT_RULE_VERSION,
ALERT_SEVERITY,
ALERT_START,
ALERT_TIME_RANGE,
ALERT_SYSTEM_STATUS,
ALERT_UUID,
ECS_VERSION,
EVENT_ACTION,
EVENT_KIND,
EVENT_MODULE,
KIBANA_NAMESPACE,
ALERT_RULE_UUID,
ALERT_RULE_CATEGORY,
ALERT_THREAT_FRAMEWORK,
ALERT_THREAT_TACTIC_ID,
ALERT_THREAT_TACTIC_NAME,
Expand All @@ -262,8 +234,6 @@ export {
ALERT_SUPPRESSION_DOCS_COUNT,
TAGS,
TIMESTAMP,
SPACE_IDS,
VERSION,
};

export type TechnicalRuleDataFieldName = ValuesType<typeof fields & typeof namespaces>;
Loading

0 comments on commit b7a0204

Please sign in to comment.