8.16 Elastic Security Release Notes

docs/release-notes.asciidoc

@@ -3,6 +3,7 @@
 
 This section summarizes the changes in each release.
 
+* <<release-notes-8.16.0, {elastic-sec} version 8.16.0>>
 * <<release-notes-8.15.3, {elastic-sec} version 8.15.3>>
 * <<release-notes-8.15.2, {elastic-sec} version 8.15.2>>
 * <<release-notes-8.15.1, {elastic-sec} version 8.15.1>>
@@ -65,6 +66,7 @@ This section summarizes the changes in each release.
 * <<release-notes-8.0.0, {elastic-sec} version 8.0.0>>
 * <<release-notes-8.0.0-rc2, {elastic-sec} version 8.0.0-rc2>>
 
+include::release-notes/8.16.asciidoc[]
 include::release-notes/8.15.asciidoc[]
 include::release-notes/8.14.asciidoc[]
 include::release-notes/8.13.asciidoc[]

docs/release-notes/8.16.asciidoc

@@ -0,0 +1,155 @@
+[[release-notes-header-8.16.0]]
+== 8.16
+
+[discrete]
+[[release-notes-8.16.0]]
+=== 8.16.0
+
+[discrete]
+[[known-issue-8.16.0]]
+==== Known issues
+
+// tag::known-issue[]
+[discrete]
+.Duplicate alerts can be produced from manually running threshold rules 
+[%collapsible]
+====
+*Details* +
+On November 12, 2024, it was discovered that manually running threshold rules could produce duplicate alerts if the date range was already covered by a scheduled rule execution.
+
+====
+// end::known-issue[]
+
+// tag::known-issue[]
+[discrete]
+.Attempting to edit an Elastic AI Assistant Knowledge Base index results in an error 
+[%collapsible]
+====
+*Details* +
+Updating a Knowledge Base entry of type "index" results in an error.
+
+*Workaround* +
+Instead of updating an "index" entry, delete it and add it again with the desired changes.
+
+====
+// end::known-issue[]
+
+
+// tag::known-issue[]
+[discrete]
+.Manually running custom query rules with suppression could suppress more alerts than expected
+[%collapsible]
+====
+*Details* +
+On November 12, 2024, it was discovered that manually running a custom query rule with suppression could incorrectly inflate the number of suppressed alerts. 
+
+====
+// end::known-issue[]
+
+// tag::known-issue-189676[]
+[discrete]
+.Tags appear in Elastic AI Assistant's responses
+[%collapsible]
+====
+*Details* +
+On August 1, 2024, it was discovered that Elastic AI Assistant's responses when using Bedrock Sonnet 3.5 may include `<antThinking>` tags, for example `<search_quality_reflection>` ({kibana-issue}189676[#189676]).
+
+====
+// end::known-issue-189676[]
+
+[discrete]
+[[breaking-changes-8.16.0]]
+==== Breaking changes
+
+* During shutdown, {kib} now waits for all the ongoing requests to complete according to the `server.shutdownTimeout` setting. During that period, the incoming socket is closed and any new incoming requests are rejected. Before this update, new incoming requests received a response with the status code 503 and body { "message": "{kib} is shutting down and not accepting new incoming requests" }.
+
+[discrete]
+[[features-8.16.0]]
+==== New features
+
+// * Introduces a new API route for listing Entity Store entities: `GET /api/entity_store/entities/list` ({kibana-pull}192806[#192806]).
+* Introduces Knowledge Base for Elastic AI Assistant, which allows you to specify information for AI Assistant to remember when responding to your queries ({kibana-pull}186566[#186566], {kibana-pull}192665[#192665]).
+* Enables detection rules to automatically execute system actions such as opening a case ({kibana-pull}183937[#183937]). 
+* Adds syntax validation for {esql} queries ({kibana-pull}189780[#189780]).
+* Provides a way to view {es} queries that run during rule execution. This option is provided for {esql} and EQL rules only ({kibana-pull}191107[#191107]).
+* Allows you to create and update a rule even when some data-related validation errors are present in the query field ({kibana-pull}191487[#191487]).
+* Introduces a new advanced setting `securitySolution:enableVisualizationsInFlyout`. When enabled, you can examine alerts and events in the **Visualize** tab, which provides a more detailed view of the event analyzer and Session View ({kibana-pull}194012[#194012], {kibana-pull}192531[#192531], {kibana-pull}192643[#192643]).
+* Creates a new advanced setting `securitySolution:excludedDataTiersForRuleExecution` that allows you to exclude cold and frozen data from rule execution. This setting does not apply to {ml} rules ({kibana-pull}186908[#186908]). 
+* Introduces the ability to add notes to alerts and events, in addition to Timeline ({kibana-pull}186787[#186787], {kibana-pull}186807[#186807], {kibana-pull}186931[#186931], {kibana-pull}186946[#186946], {kibana-pull}187214[#187214], {kibana-pull}193373[#193373]).
+//PR for notes feature is incoming.
+* Enhances the Insights in the section in the alert and event details flyouts by providing available misconfiguration and vulnerabilities findings ({kibana-pull}195509[#195509]).
+* Removes the `securitySolution:enableAssetCriticality` advanced setting and enables <<asset-criticality, asset criticality>> workflows by default ({kibana-pull}196270[#196270])
+* Adds RBAC for Elastic AI Assistant's knowledge base ({kibana-pull}195733[#195733]).
+* Adds RBAC for Attack Discovery ({kibana-pull}188788[#188788]).
+* Adds historical results to the Data Quality dashboard and updates its UI ({kibana-pull}191898[#191898], {kibana-pull}196127[#196127]). 
+* Enables agentless deployment for Elastic's Cloud Security Posture Management integration ({kibana-pull}191557[#191557]).
+* The host field size reduction setting on {elastic-defend}'s integration policy is now turned off by default. To turn it on, configure the `[os].advanced.set_extended_host_information` <<adv-policy-settings,advanced policy setting>>.
+* To reduce CPU usage, I/O, and event sizes, you can turn on process event aggregation when configuring your {elastic-defend} integration policy. Related process events that occur in rapid succession are combined into fewer aggregate events. To turn on process event aggregation, configure the `advanced.events.aggregate_process` <<adv-policy-settings,advanced policy setting>>.
+* To reduce CPU usage, I/O, and event sizes, you can now turn off of MD5, SHA-1, and SHA-256 hashes in events when configuring your {elastic-defend} integration policy. Example fields include `process.hash.md5` and `file.hash.sha1`. 
+* You can now configure your {elastic-defend} integration policy to allow the collection of SHA-256 file hashes in file events. Before doing so, consider the following caveats: 
+** This can greatly increase {elastic-defend}'s CPU and I/O utilization, and impact system responsiveness.
+** This can significantly delay event enrichment, and lead to Behavioral Protection rules firing too late to effectively stop malicious behavior.
+** This can cause event processing queues to overflow, and lead to dropped events.
+** Many file events won't contain hashes. Hash collection is best effort and not guaranteed to be present in every event. Hashes are collected asynchronously, and shortly after the file activity. Hashes might be missing if the file was rapidly renamed, moved, deleted, or (on Windows) opened by another process without https://learn.microsoft.com/en-us/windows/win32/fileio/creating-and-opening-files[read sharing].
+* Improves {elastic-defend} enabling the use of dynamic {filebeat-ref}/kafka-output.html#topic-option-kafka[topics] for the kafka output.
+* Improves {elastic-defend} by integrating a new Event Tracing for Windows (ETW) provider (Microsoft-Windows-WMI-Activity) to create new event types that can be used by prebuilt endpoint rules to detect malicious WMI activity.
+
+[discrete]
+[[enhancements-8.16.0]]
+==== Enhancements
+* Enables you to open the Rule details flyout from the Alerts table ({kibana-pull}191764[#191764]).
+* Allows you to resize the alert and event details flyouts and choose how it's displayed (over the Alerts table or next to it) ({kibana-pull}192906[#192906], {kibana-pull}182615[#182615]).
+* Allows you to enable entity risk scoring in multiple {kib} spaces ({kibana-pull}192671[#192671]).
+* Creates a new API endpoint for cleaning up entity risk scoring data: `DELETE /api/risk_score/engine/dangerously_delete_data` ({kibana-pull}191843[#191843], {kibana-pull}189872[#189872]). 
+* Allows Automatic Import to analyze a larger number of sample events when generating a new integration ({kibana-pull}196233[#196233]).
+* Allows Automatic Import to recognize CSV logs and create integrations for CSV data ({kibana-pull}196228[#196228], {kibana-pull}194386[#194386]).
+* Removes Elastic AI Assistant's default system prompts. The instructions previously contained in those prompts are now automatically included without user interaction, so Elastic AI Assistant will remain focused on relevant topics. Custom system prompts are still available ({kibana-pull}191847[#191847]). 
+* Improves Elastic AI Assistant's ability to generate {esql} queries ({kibana-pull}195480[#195480], {kibana-pull}188492[#188492]). 
+* Adds a button that lets you quickly add queries generated by Elastic AI Assistant to a rule's definition ({kibana-pull}190963[#190963]).
+* Adds an "Other" option to the OpenAI connector's "Select an OpenAI provider" dropdown menu. Select this option when <<connect-to-byo-llm, connecting to your own custom LLM>> ({kibana-pull}194831[#194831]).
+* Improves Attack Discovery in the following ways ({kibana-pull}195669[#195669]): 
+** Attack Discovery can now process up to 500 alerts (previous maximum: 100). This setting can now be adjusted directly from the Attack Discovery page, and is stored locally instead of in {es}
+** Attack Discovery now combines related discoveries that would previously have appeared separately
+** Attack Discovery now detects and displays an error instead of hallucinated output
+* Adds an **Install and enable** button to the **Add Elastic Rules** page, so you don't have to wait for rules to install before enabling them ({kibana-pull}191529[#191529]). 
+* Updates the Get Started tour for {elastic-sec} ({kibana-pull}192247[#192247]). 
+* Improves loading performance for various pages in {kib} ({kibana-pull}194241[#194241]). 
+* Adds Alert Suppression and Investigation Fields to the rule upgrade workflow ({kibana-pull}195499[#195499]). 
+* Adds User and Global Artifacts to the {fleet} Policy Response flyout and to the Endpoint details flyout ({kibana-pull}184125[#184125]). 
+* Allows you to recalculate entity risk scores immediately after you upload asset criticality data ({kibana-pull}187577[#187577]). 
+* Adds a {kib} advanced setting `securitySolution:maxUnassociatedNotes`, which allows you to set the maximum number of notes that can be attached to alerts and events ({kibana-pull}194947[#194947]). 
+* Adds the `IS` operator as an option when configuring a Windows signature blocklist entry ({kibana-pull}190515[#190515]). 
+* Improves network previews in the alert details flyout ({kibana-pull}190560[#190560]). 
+* Enhances {elastic-defend} by improving the `call_stack_final_user_module` attribution where potential `proxy_call` modules are encountered during Windows call stack analysis.
+* Adds new fields to {elastic-defend} API events to improve context for the triage of Behavior Alerts. The new `call_stack_final_user_module` fields are `allocation_private_bytes`, `protection`, `protection_provenance_path`, and `reason`.
+* Adds a new {elastic-defend} API event for https://learn.microsoft.com/en-us/windows/win32/api/ioapiset/nf-ioapiset-deviceiocontrol[`DeviceIoControl`] calls to support the detection of driver abuse. This feature is only supported on Windows 11 Desktop versions.
+* Ensures security artifacts are updated when the {elastic-defend} service starts.
+* Improves error messages that are returned when {elastic-defend} receives invalid or unsupported cryptographic keys from the {elastic-defend} policy. 
+* Ensures that {elastic-defend} tells {fleet} that it's `orphaned` if the connection between {elastic-defend} and {agent} stops for an extended period of time. {fleet} uses this information to provide you with additional troubleshooting context.
+* Adds SOCKS5 proxy support to {elastic-defend}'s {ls} output.
+* Ensures that on Windows, {elastic-defend} uses https://www.elastic.co/security-labs/finding-truth-in-the-shadows[Intel CET and AMD Shadow Stacks] to collect call stacks, where supported. This improves performance and enables the detection of certain defense evasions. You can turn this feature off in {elastic-defend} <<adv-policy-settings,advanced policy settings>> ({kibana-pull}190553[#190553]).
+* Restore {elastic-defend}'s support for Windows Server 2012, which was removed in 8.13.0.
+* Improves {elastic-defend}'s caching to reduce memory usage on Windows.
+* Enhances {elastic-defend} by reducing the size of process events, which reduces excessive process ancestry entries and shortens the entity ID.
+* Improves the reliability and system resource usage of {elastic-defend}'s Windows network driver.
+
+[discrete]
+[[bug-fixes-8.16.0]]
+==== Bug fixes
+
+* Prevents an empty warning message from appearing for rule executions ({kibana-pull}186096[#186096]). 
+* Fixes an error that could occur during rule execution when the source index had a non-ECS-compliant text field ({kibana-pull}187673[#187673]). 
+* Removes unnecessary empty space below the title of the Open Timeline modal ({kibana-pull}188837[#188837]). 
+* Improves the Alerts table's performance ({kibana-pull}192827[#192827]). 
+* Removes the requirement that you have unnecessary {kib} {fleet} privileges to access some cloud security posture Findings ({kibana-pull}194069[#194069]).
+* Fixes an issue that could cause fields for all indices to appear when you tried to add a rule filter ({kibana-pull}194678[#194678]).
+* Fixes an {elastic-defend} bug where network event deduplication logic could incorrectly drop Linux network events.
+* Fixes an {elastic-defend} bug where Windows API events might be dropped if they contain Unicode characters that can't be converted to ANSI.
+* Ensures that {elastic-defend} does not emit an empty `memory_region` if it can't enrich a memory region in an API event. With this fix, {elastic-defend} removes these fields.
+* Fixes a bug where {elastic-defend} could fail to properly enrich Windows API events for short-lived processes on older operating systems that don't natively include this telemetry, such as Windows Server 2019. This might result in dropped or unattributed API events. 
+* Fixes a bug that prevented host name uniformity with {beats} products. If you request for {elastic-defend} use the fully qualified domain name (FQDN) in the `host.name` field, {elastic-defend} now reports the FQDN exactly as the OS reports it, instead of lowercasing by default.
+* Fixes an {elastic-defend} bug in behavior protection alerts, where prevention alerts could mistakenly be labeled as detection alerts.
+* Fixes a bug that caused {elastic-defend} to crash if a Kafka connection is busy.
+* Fixes cases where Automatic Import could generate invalid processors containing array access ({kibana-pull}196207[#196207]).
+* Improves Timeline's table performance when row renderers are switched on ({kibana-pull}193316[#193316]).
+* Fixes misaligned filter control labels on the Alerts page ({kibana-pull}192094[#192094]).