Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[8.16] Updates docs for and related to the excludedDataTiersForRuleExecution advanced setting #5962

Merged
merged 38 commits into from
Nov 14, 2024
Merged
Show file tree
Hide file tree
Changes from 6 commits
Commits
Show all changes
38 commits
Select commit Hold shift + click to select a range
85bd755
First draft
nastasha-solomon Oct 19, 2024
1fff051
Updating IM rules
nastasha-solomon Oct 19, 2024
977bdfb
Merge branch 'main' into issue-5925-cold-frozen-data-tiers
nastasha-solomon Oct 27, 2024
f51d3ae
Merge branch 'main' into issue-5925-cold-frozen-data-tiers
nastasha-solomon Nov 12, 2024
4f22e44
disclaimer about certain rule types and shards
nastasha-solomon Nov 12, 2024
0c9a44a
Minor tweak to dsl query docs
nastasha-solomon Nov 12, 2024
e4cac58
Update docs/detections/detection-engine-intro.asciidoc
nastasha-solomon Nov 13, 2024
b9c4739
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 13, 2024
479de19
Update docs/getting-started/advanced-setting.asciidoc
nastasha-solomon Nov 13, 2024
de1b7a3
Update docs/release-notes/8.16.asciidoc
nastasha-solomon Nov 13, 2024
203b0bb
Marshall's suggestion
nastasha-solomon Nov 13, 2024
b5b1a30
Merge branch 'main' into issue-5925-cold-frozen-data-tiers
nastasha-solomon Nov 13, 2024
ae1c0f8
Update docs/detections/detection-engine-intro.asciidoc
nastasha-solomon Nov 13, 2024
76ba696
Removes note that's no longer needed
nastasha-solomon Nov 13, 2024
c345d01
Moves file back to remove this change from the PR
nastasha-solomon Nov 13, 2024
8946d5b
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 13, 2024
000a858
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 13, 2024
6551414
Updates what's new
nastasha-solomon Nov 13, 2024
cbc62e1
Fixed title
nastasha-solomon Nov 13, 2024
c9bc108
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 13, 2024
6bc8cd9
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 13, 2024
ebe4048
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 13, 2024
2c30b2c
Update docs/getting-started/advanced-setting.asciidoc
nastasha-solomon Nov 13, 2024
e23b060
Update docs/getting-started/advanced-setting.asciidoc
nastasha-solomon Nov 13, 2024
e6bc067
Update docs/whats-new.asciidoc
nastasha-solomon Nov 13, 2024
8c46fcc
Update docs/whats-new.asciidoc
nastasha-solomon Nov 13, 2024
7c216dd
Update docs/release-notes/8.16.asciidoc
nastasha-solomon Nov 13, 2024
f0c8925
Fixes a typo
nastasha-solomon Nov 13, 2024
3e4bf11
Minor wording adjustments
nastasha-solomon Nov 13, 2024
122da41
Update docs/whats-new.asciidoc
nastasha-solomon Nov 14, 2024
8e6e307
Update docs/release-notes/8.16.asciidoc
nastasha-solomon Nov 14, 2024
13a8bd9
Update docs/getting-started/advanced-setting.asciidoc
nastasha-solomon Nov 14, 2024
74c640f
Merge branch 'main' into issue-5925-cold-frozen-data-tiers
nastasha-solomon Nov 14, 2024
c633189
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 14, 2024
aa2b0eb
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 14, 2024
c466d0d
Update docs/detections/detections-exclude-cold-frozen-data-tiers.asci…
nastasha-solomon Nov 14, 2024
64be6ff
Merge branch 'main' into issue-5925-cold-frozen-data-tiers
nastasha-solomon Nov 14, 2024
2d0aca5
Update docs/detections/detection-engine-intro.asciidoc
nastasha-solomon Nov 14, 2024
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion docs/detections/detection-engine-intro.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -86,7 +86,9 @@ Indicator match rules provide a powerful capability to search your security data

In addition, the following support restrictions are in place:

* {elastic-sec} does not support the use of either cold or frozen {ref}/data-tiers.html[tier data] with indicator match rules.
* Indicator match rules don't support cold or frozen data, but will query cold and frozen {ref}/data-tiers.html[data tiers] if they exist. To exclude query results from cold and frozen tiers, configure the `excludedDataTiersForRuleExecution` <<exclude-cold-frozen-data-rule-executions,advanced setting>> (which applies to all rules in a space), or add a <<exclude-cold-frozen-data-individual-rules,Query DSL filter>> to individual rules.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
+
IMPORTANT: When the `excludedDataTiersForRuleExecution` advanced setting is enabled, some rule types (specifically the indicator match, event correlation, and {esql} rule types) might fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. The best path forward continues to be modifying the index patterns to only use hot tier data.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
* Indicator match rules with an additional look-back time value greater than 24 hours are not supported.

[float]
Expand Down
Original file line number Diff line number Diff line change
@@ -1,18 +1,24 @@
[[exclude-cold-frozen-data-individual-rules]]
== Exclude cold and frozen data from a rule
== Exclude cold and frozen data from rule executions
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

:frontmatter-description: Configure a rule to ignore cold and frozen data during execution.
:frontmatter-tags-products: [security]
:frontmatter-tags-content-type: [how-to]
:frontmatter-tags-user-goals: [manage]

Rules that query cold and frozen data might perform more slowly. To exclude cold and frozen data, add a Query DSL filter that ignores cold and frozen {ref}/data-tiers.html[data tiers] when executing. You can add the filter when creating a new rule or updating an existing one.
Rules that query cold and frozen {ref}/data-tiers.html[data tiers] might perform more slowly. To exclude query results from cold and frozen tiers, add a Query DSL filter that ignores cold and frozen documents when executing. This can help Elasticsearch exclude cold and frozen data more efficiently. You can add the filter when creating a new rule or updating an existing one.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

NOTE: This method is not supported for {esql} and {ml} rules.
TIP: To ensure that _all_ rules in a {kib} space exclude cold and frozen documents when executing, configure the `excludedDataTiersForRuleExecution` <<exclude-cold-frozen-data-rule-executions,advanced setting>>.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

TIP: To ensure that _all_ rules in a {kib} space exclude cold and frozen data when executing, configure the `excludedDataTiersForRuleExecution` <<exclude-cold-frozen-data-rule-executions,advanced setting>>.
[IMPORTANT]
====

Here is a sample Query DSL filter that excludes frozen tier data from a rule's execution:
* This method is not supported for {esql} and {ml} rules.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
* Some rule types (specifically the indicator match, event correlation, and {esql} rule types) might fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. The best path forward continues to be modifying the index patterns to only use hot tier data.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

====

Here is a sample Query DSL filter that excludes documents from a frozen tier during a rule's execution:
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

[source,console]
----
Expand All @@ -29,7 +35,7 @@ Here is a sample Query DSL filter that excludes frozen tier data from a rule's e
}
----

Here is another sample Query DSL filter that excludes cold and frozen tier data from a rule's execution:
Here is another sample Query DSL filter that excludes documents from cold and frozen tiers during a rules execution:
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

[source,console]
----
Expand Down
6 changes: 4 additions & 2 deletions docs/getting-started/advanced-setting.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -189,7 +189,7 @@ The `securitySolution:maxUnassociatedNotes` field determines the maximum number
[[exclude-cold-frozen-data-rule-executions]]
== Exclude cold and frozen data from rule executions
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

To ensure rules don't search cold and frozen data when executing, specify cold and frozen {ref}/data-tiers.html[data tiers] in the `excludedDataTiersForRuleExecution` field. Multiple data tiers must be separated by commas, for example: `data_frozen`, `data_cold`. This setting is turned off by default; turning it on can improve rule performance and reduce execution time.
To ensure rules exclude query results from cold and frozen tiers when executing, specify cold and frozen {ref}/data-tiers.html[data tiers] in the `excludedDataTiersForRuleExecution` field. Multiple data tiers must be separated by commas, for example: `data_frozen`, `data_cold`. This setting is turned off by default; turning it on can improve rule performance and reduce execution time.

This setting does not apply to {ml} rules.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved

Expand All @@ -198,4 +198,6 @@ This setting does not apply to {ml} rules.

This setting applies to all rules in a {kib} space. To only exclude cold and frozen data from specific rules, add a <<exclude-cold-frozen-data-individual-rules,Query DSL filter>> to the rules you want affected.

====
====

IMPORTANT: When the `excludedDataTiersForRuleExecution` advanced setting is enabled, some rule types (specifically the indicator match, event correlation, and {esql} rule types) might fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. The best path forward continues to be modifying the index patterns to only use hot tier data.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
2 changes: 2 additions & 0 deletions docs/release-notes/8.16.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -83,6 +83,8 @@ On November 12, 2024, it was discovered that manually running a custom query rul
* Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]).
* Introduces a new advanced setting, `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]).
* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]).
+
IMPORTANT: When the `excludedDataTiersForRuleExecution` advanced setting is enabled, some rule types (specifically the indicator match, event correlation, and {esql} rule types) might fail if a frozen or cold shard that matches the rule's specified index pattern is unavailable during rule executions. The best path forward continues to be modifying the index patterns to only use hot tier data.
nastasha-solomon marked this conversation as resolved.
Show resolved Hide resolved
* Enhances the Insights section of the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]).
* Turns off the host field size reduction setting on {elastic-defend}'s integration policy by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <<adv-policy-settings,advanced policy setting>>.
* Allows you to reduce CPU usage, I/O, and event sizes by turning on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <<adv-policy-settings,advanced policy setting>>.
Expand Down