Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Security Solution][Alerts] Format alerts for per-alert action context variables #155812

Closed
ymao1 opened this issue Apr 25, 2023 · 6 comments · Fixed by #155829
Closed

[Security Solution][Alerts] Format alerts for per-alert action context variables #155812

ymao1 opened this issue Apr 25, 2023 · 6 comments · Fixed by #155829
Assignees
Labels
bug Fixes for quality problems that affect the customer experience Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.8.0

Comments

@ymao1
Copy link
Contributor

ymao1 commented Apr 25, 2023

In this PR, detection rules were switched to support per-alert actions. When passing the context variable, it was suggested that we should be calling formatAlert to format the alert for notifications, however doing that causes some test failures because formatAlert is fairly heavyweight and the following tests were timing out:

We should (1) determine if formatAlert is strictly necessary for these context variables, (2) determine if we can move this call to the alerting framework, and (3) if not, can we increase the test timeout for the failing tests?

@botelastic botelastic bot added the needs-team Issues missing a team label label Apr 25, 2023
@ymao1 ymao1 added bug Fixes for quality problems that affect the customer experience Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.8.0 and removed needs-team Issues missing a team label labels Apr 25, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/response-ops (Team:ResponseOps)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@ymao1 ymao1 changed the title [Security Solution][Alerts] [Security Solution][Alerts] Format alerts for per-alert action context variables Apr 25, 2023
@ymao1
Copy link
Contributor Author

ymao1 commented Apr 25, 2023

Alert prior to formatAlert:

{
	"_id": "c32c2c39accb0650e02e9770b0b7c35111812a7308d8e91305468d058ca4d501",
	"_index": ".internal.alerts-security.alerts-default-000001",
	"kibana.alert.start": "2023-04-25T23:43:08.029Z",
	"kibana.alert.last_detected": "2023-04-25T23:43:08.029Z",
	"kibana.version": "8.8.0",
	"kibana.alert.rule.parameters": {
		"description": "test",
		"risk_score": 21,
		"severity": "low",
		"license": "",
		"meta": {
			"from": "1m",
			"kibana_siem_app_url": "https://localhost:5601/app/security"
		},
		"author": [],
		"false_positives": [],
		"from": "now-360s",
		"rule_id": "06a766ae-46fa-456a-8f48-3556e18992c6",
		"max_signals": 100,
		"risk_score_mapping": [],
		"severity_mapping": [],
		"threat": [],
		"to": "now",
		"references": [],
		"version": 1,
		"exceptions_list": [],
		"immutable": false,
		"related_integrations": [],
		"required_fields": [],
		"setup": "",
		"type": "query",
		"language": "kuery",
		"index": [".kibana-event-log*"],
		"query": "event.provider:\"alerting\" ",
		"filters": []
	},
	"kibana.alert.rule.category": "Custom Query Rule",
	"kibana.alert.rule.consumer": "siem",
	"kibana.alert.rule.execution.uuid": "72a7e990-46c7-4372-8205-0003f4149982",
	"kibana.alert.rule.name": "test",
	"kibana.alert.rule.producer": "siem",
	"kibana.alert.rule.revision": 0,
	"kibana.alert.rule.rule_type_id": "siem.queryRule",
	"kibana.alert.rule.uuid": "eb777af0-e3c2-11ed-b45d-29e737a42ba7",
	"kibana.space_ids": ["default"],
	"kibana.alert.rule.tags": [],
	"@timestamp": "2023-04-25T23:43:08.009Z",
	"message": "example.always-firing:bff70170-e3c2-11ed-b45d-29e737a42ba7: 'data' alert '8eece59d-7ef5-48b2-9e80-f514ceb624be' has recovered",
	"rule": {
		"id": "bff70170-e3c2-11ed-b45d-29e737a42ba7",
		"license": "basic",
		"category": "example.always-firing",
		"ruleset": "AlertingExample",
		"name": "data"
	},
	"ecs": {
		"version": "1.8.0"
	},
	"event.provider": "alerting",
	"event.action": "recovered-instance",
	"event.kind": "signal",
	"event.category": ["AlertingExample"],
	"event.start": "2023-04-25T23:42:40.788Z",
	"event.end": "2023-04-25T23:42:52.804Z",
	"event.duration": "12016000000",
	"kibana.alert.original_time": "2023-04-25T23:42:52.804Z",
	"kibana.alert.ancestors": [{
		"id": "SOvMuocBsbEE7DT-c8sI",
		"type": "event",
		"index": ".kibana-event-log-8.8.0-000001",
		"depth": 0
	}],
	"kibana.alert.status": "active",
	"kibana.alert.workflow_status": "open",
	"kibana.alert.depth": 1,
	"kibana.alert.reason": "AlertingExample event created low alert test.",
	"kibana.alert.severity": "low",
	"kibana.alert.risk_score": 21,
	"kibana.alert.rule.actions": [{
		"group": "default",
		"id": "slack",
		"params": {
			"message": "Rule {{context.rule.name}} generated alert {{alert.id}}\n\n{{#context.alerts}}{{.}}{{/context.alerts}}"
		},
		"action_type_id": ".slack",
		"uuid": "2eb98d7c-5f9e-4e29-9ad6-7025ec22b403",
		"frequency": {
			"notifyWhen": "onActiveAlert",
			"throttle": null,
			"summary": false
		}
	}],
	"kibana.alert.rule.author": [],
	"kibana.alert.rule.created_at": "2023-04-25T23:43:03.666Z",
	"kibana.alert.rule.created_by": "elastic",
	"kibana.alert.rule.description": "test",
	"kibana.alert.rule.enabled": true,
	"kibana.alert.rule.exceptions_list": [],
	"kibana.alert.rule.false_positives": [],
	"kibana.alert.rule.from": "now-360s",
	"kibana.alert.rule.immutable": false,
	"kibana.alert.rule.interval": "5m",
	"kibana.alert.rule.indices": [".kibana-event-log*"],
	"kibana.alert.rule.license": "",
	"kibana.alert.rule.max_signals": 100,
	"kibana.alert.rule.references": [],
	"kibana.alert.rule.risk_score_mapping": [],
	"kibana.alert.rule.rule_id": "06a766ae-46fa-456a-8f48-3556e18992c6",
	"kibana.alert.rule.severity_mapping": [],
	"kibana.alert.rule.threat": [],
	"kibana.alert.rule.to": "now",
	"kibana.alert.rule.type": "query",
	"kibana.alert.rule.updated_at": "2023-04-25T23:43:03.666Z",
	"kibana.alert.rule.updated_by": "elastic",
	"kibana.alert.rule.version": 1,
	"kibana.alert.url": "https://localhost:5601/app/security/alerts/redirect/c32c2c39accb0650e02e9770b0b7c35111812a7308d8e91305468d058ca4d501?index=.alerts-security.alerts-default&timestamp=2023-04-25T23:43:08.009Z",
	"kibana.alert.uuid": "c32c2c39accb0650e02e9770b0b7c35111812a7308d8e91305468d058ca4d501",
	"kibana.alert.rule.meta.from": "1m",
	"kibana.alert.rule.meta.kibana_siem_app_url": "https://localhost:5601/app/security",
	"kibana.alert.rule.risk_score": 21,
	"kibana.alert.rule.severity": "low",
	"kibana.alert.original_event.provider": "alerting",
	"kibana.alert.original_event.action": "recovered-instance",
	"kibana.alert.original_event.kind": "alert",
	"kibana.alert.original_event.category": ["AlertingExample"],
	"kibana.alert.original_event.start": "2023-04-25T23:42:40.788Z",
	"kibana.alert.original_event.end": "2023-04-25T23:42:52.804Z",
	"kibana.alert.original_event.duration": "12016000000"
}

Alert after formatAlert:

{
	"signal": {
		"depth": 1,
		"original_event": {
			"action": "recovered-instance",
			"category": ["AlertingExample"],
			"duration": "12016000000",
			"end": "2023-04-25T23:42:52.804Z",
			"kind": "alert",
			"provider": "alerting",
			"start": "2023-04-25T23:42:40.788Z"
		},
		"original_time": "2023-04-25T23:42:52.804Z",
		"reason": "AlertingExample event created low alert test.",
		"rule": {
			"author": [],
			"created_at": "2023-04-25T23:43:03.666Z",
			"created_by": "elastic",
			"description": "test",
			"enabled": true,
			"false_positives": [],
			"from": "now-360s",
			"id": "eb777af0-e3c2-11ed-b45d-29e737a42ba7",
			"immutable": false,
			"interval": "5m",
			"license": "",
			"max_signals": 100,
			"name": "test",
			"references": [],
			"risk_score": 21,
			"rule_id": "06a766ae-46fa-456a-8f48-3556e18992c6",
			"severity": "low",
			"tags": [],
			"to": "now",
			"type": "query",
			"updated_at": "2023-04-25T23:43:03.666Z",
			"updated_by": "elastic",
			"version": 1
		},
		"status": "open"
	},
	"_id": "c32c2c39accb0650e02e9770b0b7c35111812a7308d8e91305468d058ca4d501",
	"_index": ".internal.alerts-security.alerts-default-000001",
	"kibana": {
		"alert": {
			"start": "2023-04-25T23:43:08.029Z",
			"last_detected": "2023-04-25T23:43:08.029Z",
			"rule": {
				"parameters": {
					"description": "test",
					"risk_score": 21,
					"severity": "low",
					"license": "",
					"meta": {
						"from": "1m",
						"kibana_siem_app_url": "https://localhost:5601/app/security"
					},
					"author": [],
					"false_positives": [],
					"from": "now-360s",
					"rule_id": "06a766ae-46fa-456a-8f48-3556e18992c6",
					"max_signals": 100,
					"risk_score_mapping": [],
					"severity_mapping": [],
					"threat": [],
					"to": "now",
					"references": [],
					"version": 1,
					"exceptions_list": [],
					"immutable": false,
					"related_integrations": [],
					"required_fields": [],
					"setup": "",
					"type": "query",
					"language": "kuery",
					"index": [".kibana-event-log*"],
					"query": "event.provider:\"alerting\" ",
					"filters": []
				},
				"category": "Custom Query Rule",
				"consumer": "siem",
				"execution": {
					"uuid": "72a7e990-46c7-4372-8205-0003f4149982"
				},
				"name": "test",
				"producer": "siem",
				"revision": 0,
				"rule_type_id": "siem.queryRule",
				"uuid": "eb777af0-e3c2-11ed-b45d-29e737a42ba7",
				"tags": [],
				"actions": [{
					"group": "default",
					"id": "slack",
					"params": {
						"message": "Rule {{context.rule.name}} generated alert {{alert.id}}\n\n{{#context.alerts}}{{.}}{{/context.alerts}}"
					},
					"action_type_id": ".slack",
					"uuid": "2eb98d7c-5f9e-4e29-9ad6-7025ec22b403",
					"frequency": {
						"notifyWhen": "onActiveAlert",
						"throttle": null,
						"summary": false
					}
				}],
				"author": [],
				"created_at": "2023-04-25T23:43:03.666Z",
				"created_by": "elastic",
				"description": "test",
				"enabled": true,
				"exceptions_list": [],
				"false_positives": [],
				"from": "now-360s",
				"immutable": false,
				"interval": "5m",
				"indices": [".kibana-event-log*"],
				"license": "",
				"max_signals": 100,
				"references": [],
				"risk_score_mapping": [],
				"rule_id": "06a766ae-46fa-456a-8f48-3556e18992c6",
				"severity_mapping": [],
				"threat": [],
				"to": "now",
				"type": "query",
				"updated_at": "2023-04-25T23:43:03.666Z",
				"updated_by": "elastic",
				"version": 1,
				"meta": {
					"from": "1m",
					"kibana_siem_app_url": "https://localhost:5601/app/security"
				},
				"risk_score": 21,
				"severity": "low"
			},
			"original_time": "2023-04-25T23:42:52.804Z",
			"ancestors": [{
				"id": "SOvMuocBsbEE7DT-c8sI",
				"type": "event",
				"index": ".kibana-event-log-8.8.0-000001",
				"depth": 0
			}],
			"status": "active",
			"workflow_status": "open",
			"depth": 1,
			"reason": "AlertingExample event created low alert test.",
			"severity": "low",
			"risk_score": 21,
			"url": "https://localhost:5601/app/security/alerts/redirect/c32c2c39accb0650e02e9770b0b7c35111812a7308d8e91305468d058ca4d501?index=.alerts-security.alerts-default&timestamp=2023-04-25T23:43:08.009Z",
			"uuid": "c32c2c39accb0650e02e9770b0b7c35111812a7308d8e91305468d058ca4d501",
			"original_event": {
				"provider": "alerting",
				"action": "recovered-instance",
				"kind": "alert",
				"category": ["AlertingExample"],
				"start": "2023-04-25T23:42:40.788Z",
				"end": "2023-04-25T23:42:52.804Z",
				"duration": "12016000000"
			}
		},
		"version": "8.8.0",
		"space_ids": ["default"]
	},
	"@timestamp": "2023-04-25T23:43:08.009Z",
	"message": "example.always-firing:bff70170-e3c2-11ed-b45d-29e737a42ba7: 'data' alert '8eece59d-7ef5-48b2-9e80-f514ceb624be' has recovered",
	"rule": {
		"id": "bff70170-e3c2-11ed-b45d-29e737a42ba7",
		"license": "basic",
		"category": "example.always-firing",
		"ruleset": "AlertingExample",
		"name": "data"
	},
	"ecs": {
		"version": "1.8.0"
	},
	"event": {
		"provider": "alerting",
		"action": "recovered-instance",
		"kind": "signal",
		"category": ["AlertingExample"],
		"start": "2023-04-25T23:42:40.788Z",
		"end": "2023-04-25T23:42:52.804Z",
		"duration": "12016000000"
	}
}

@marshallmain
Copy link
Contributor

marshallmain commented Apr 26, 2023

A faster implementation of expandDottedObject in x-pack/plugins/security_solution/common/utils/expand_dotted.ts fixes the test timeout for me locally. It reduces processing time for each alert in detection engine api security and spaces enabled - Group 10 Tests involving runtime fields of source indexes and the signals index Regular runtime field mappings should copy "runtime mapping" data from a source index into the signals index in the same position when the target is ECS compatible from ~11 seconds to 0.5 seconds. The test data in that case has 7000 fields which are all being processed in the alert formatting process.

Implementation:

/*
 * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
 * or more contributor license agreements. Licensed under the Elastic License
 * 2.0; you may not use this file except in compliance with the Elastic License
 * 2.0.
 */

import { setWith } from 'lodash';

/*
 * Expands an object with "dotted" fields to a nested object with unflattened fields.
 *
 * Example:
 *   expandDottedObject({
 *     "kibana.alert.depth": 1,
 *     "kibana.alert.ancestors": [{
 *       id: "d5e8eb51-a6a0-456d-8a15-4b79bfec3d71",
 *       type: "event",
 *       index: "signal_index",
 *       depth: 0,
 *     }],
 *   })
 *
 *   => {
 *     kibana: {
 *       alert: {
 *         ancestors: [
 *           id: "d5e8eb51-a6a0-456d-8a15-4b79bfec3d71",
 *           type: "event",
 *           index: "signal_index",
 *           depth: 0,
 *         ],
 *         depth: 1,
 *       },
 *     },
 *   }
 */
export const expandDottedObject = (dottedObj: object) => {
  if (Array.isArray(dottedObj)) {
    return dottedObj;
  }
  const returnObj = {};
  Object.entries(dottedObj).forEach(([key, value]) => {
    setWith(returnObj, key, value, Object);
  });
  return returnObj;
};

I also added one extra test in expand_dotted.test.ts to see what happens when fields conflict, and verified that the behavior of the test is the same before and after the implementation change:

it('overwrites earlier fields when later fields conflict', () => {
    const simpleDottedObj = {
      'kibana.test.1': 'the spice must flow',
      'kibana.test': 2,
    };
    expect(expandDottedObject(simpleDottedObj)).toEqual({
      kibana: {
        test: 2,
      },
    });
  });

cc @e40pud

@e40pud e40pud self-assigned this Apr 26, 2023
@e40pud e40pud moved this from Awaiting Triage to In Progress in AppEx: ResponseOps - Rules & Alerts Management Apr 26, 2023
e40pud added a commit to e40pud/kibana that referenced this issue Apr 26, 2023
@e40pud e40pud moved this from In Progress to In Review in AppEx: ResponseOps - Rules & Alerts Management Apr 26, 2023
e40pud added a commit that referenced this issue Apr 26, 2023
…t variables (#155829)

## Summary

Closes [#155812](#155812)

In #155384, detection rules were
switched to support per-alert actions. When passing the context
variable, it was suggested that we should be calling formatAlert to
format the alert for notifications, however doing that causes some test
failures because formatAlert is fairly heavyweight and bunch of tests
were timing out.

Thanks to @marshallmain we have this much faster `expandDottedObject`
that solves the issue with the very slow `formatAlert`.
e40pud added a commit to e40pud/kibana that referenced this issue Apr 27, 2023
…t variables (elastic#155829)

## Summary

Closes [elastic#155812](elastic#155812)

In elastic#155384, detection rules were
switched to support per-alert actions. When passing the context
variable, it was suggested that we should be calling formatAlert to
format the alert for notifications, however doing that causes some test
failures because formatAlert is fairly heavyweight and bunch of tests
were timing out.

Thanks to @marshallmain we have this much faster `expandDottedObject`
that solves the issue with the very slow `formatAlert`.

(cherry picked from commit 8f59720)
e40pud added a commit that referenced this issue Apr 27, 2023
…context variables (#155829) (#156009)

# Backport

This will backport the following commits from `main` to `8.8`:
- [[Security Solution][Alerts] Format alerts for per-alert action
context variables
(#155829)](#155829)

<!--- Backport version: 8.9.7 -->

### Questions ?
Please refer to the [Backport tool
documentation](https://github.com/sqren/backport)

<!--BACKPORT [{"author":{"name":"Ievgen
Sorokopud","email":"ievgen.sorokopud@elastic.co"},"sourceCommit":{"committedDate":"2023-04-26T16:16:41Z","message":"[Security
Solution][Alerts] Format alerts for per-alert action context variables
(#155829)\n\n## Summary\r\n\r\nCloses
[#155812](https://github.com/elastic/kibana/issues/155812)\r\n\r\nIn
#155384, detection rules
were\r\nswitched to support per-alert actions. When passing the
context\r\nvariable, it was suggested that we should be calling
formatAlert to\r\nformat the alert for notifications, however doing that
causes some test\r\nfailures because formatAlert is fairly heavyweight
and bunch of tests\r\nwere timing out.\r\n\r\nThanks to @marshallmain we
have this much faster `expandDottedObject`\r\nthat solves the issue with
the very slow
`formatAlert`.","sha":"8f597207a222f02b1c7664bc555a9f6e744bc4aa","branchLabelMapping":{"^v8.8.0$":"main","^v(\\d+).(\\d+).\\d+$":"$1.$2"}},"sourcePullRequest":{"labels":["release_note:skip","backport:skip","v8.8.0"],"number":155829,"url":"https://github.com/elastic/kibana/pull/155829","mergeCommit":{"message":"[Security
Solution][Alerts] Format alerts for per-alert action context variables
(#155829)\n\n## Summary\r\n\r\nCloses
[#155812](https://github.com/elastic/kibana/issues/155812)\r\n\r\nIn
#155384, detection rules
were\r\nswitched to support per-alert actions. When passing the
context\r\nvariable, it was suggested that we should be calling
formatAlert to\r\nformat the alert for notifications, however doing that
causes some test\r\nfailures because formatAlert is fairly heavyweight
and bunch of tests\r\nwere timing out.\r\n\r\nThanks to @marshallmain we
have this much faster `expandDottedObject`\r\nthat solves the issue with
the very slow
`formatAlert`.","sha":"8f597207a222f02b1c7664bc555a9f6e744bc4aa"}},"sourceBranch":"main","suggestedTargetBranches":[],"targetPullRequestStates":[{"branch":"main","label":"v8.8.0","labelRegex":"^v8.8.0$","isSourceBranch":true,"state":"MERGED","url":"https://github.com/elastic/kibana/pull/155829","number":155829,"mergeCommit":{"message":"[Security
Solution][Alerts] Format alerts for per-alert action context variables
(#155829)\n\n## Summary\r\n\r\nCloses
[#155812](https://github.com/elastic/kibana/issues/155812)\r\n\r\nIn
#155384, detection rules
were\r\nswitched to support per-alert actions. When passing the
context\r\nvariable, it was suggested that we should be calling
formatAlert to\r\nformat the alert for notifications, however doing that
causes some test\r\nfailures because formatAlert is fairly heavyweight
and bunch of tests\r\nwere timing out.\r\n\r\nThanks to @marshallmain we
have this much faster `expandDottedObject`\r\nthat solves the issue with
the very slow
`formatAlert`.","sha":"8f597207a222f02b1c7664bc555a9f6e744bc4aa"}}]}]
BACKPORT-->

Co-authored-by: Kibana Machine <42973632+kibanamachine@users.noreply.github.com>
@nicpenning
Copy link

We believe that this update has caused an issue with our alerting once we upgraded to 8.8.0.

We heavily rely on the fields in the alerts as we pass them along to webhooks for notification.

It seems that at least two fields were no longer available which contained the alert name and tags. Our alerts immediately were missing this data after upgrading from 8.7.1 to 8.8.0.

We were able to correct by replacing the fields we had with the proper ones. Example:

{{alertName}} changed to {{context.rule.name}}
And
{{tags}} changed to {{rule.tags}}

With all of that being said and the above is true, two things:

  1. Did we miss this in the documentation as a feature?
  2. Do we agree that this is a breaking change and needs to be documented accordingly?

@nicpenning
Copy link

Furthermore, it appears the current docs have screenshots of a field reference that we firmly believe won't work as mentioned above:
( {{alertName }} )
https://www.elastic.co/guide/en/kibana/current/create-and-manage-rules.html#defining-rules-actions-variables

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Fixes for quality problems that affect the customer experience Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.8.0
Projects
No open projects
Development

Successfully merging a pull request may close this issue.

5 participants