[Detection Engine] Adds Alert Suppression to ML Rules (#181926)

## Summary
This PR introduces Alert Suppression for ML Detection Rules. This
feature is behaviorally similar to alerting suppression for other
Detection Engine Rule types, and nearly identical to the analogous
features for EQL rules.

There are some additional UI behaviors introduced here as well, mainly
intended to cover the shortcomings discovered in
#183100. Those behaviors are:

1. Populating the suppression field list with fields from the anomaly
index(es).
1. Disabling the suppression UI if no selected ML jobs are running
(because we cannot populate the list of fields on which they'll be
suppressing).
1. Warning the user if _some_ selected ML jobs are not running (because
the list of suppression fields may be incomplete).

See screenshots below for more info.

### Intermediate Serverless Deployment
As per the "intermediate deployment" requirements for serverless, while
the schema (and declared alert SO mappings) will be extended to allow
this functionality, the user-facing features are currently hidden behind
a feature flag. Once this is merged and released, we can issue a "final"
deployment in which the feature flag is enabled, and the feature
effectively released.


## Screenshots
* Overview of new UI fields
<img width="1044" alt="Screenshot 2024-05-16 at 3 22 02 PM"
src="https://github.com/elastic/kibana/assets/657252/8c07700d-5860-4d1e-a701-eac84fc35558">
* Example of Anomaly fields in suppression combobox
<img width="881" alt="Screenshot 2024-06-06 at 5 14 17 PM"
src="https://github.com/rylnd/kibana/assets/657252/9aa6ed99-1e02-44a0-ad1b-785136510d68">
* Suppression disabled due to no jobs running
<img width="668" alt="Screenshot 2024-06-17 at 11 23 39 PM"
src="https://github.com/elastic/kibana/assets/657252/a8636a52-31bd-4579-9bcd-d59d93c26984">
* Warning due to not all jobs running
<img width="776" alt="Screenshot 2024-06-17 at 11 26 16 PM"
src="https://github.com/elastic/kibana/assets/657252/f44c2400-570e-4fde-adce-e5841a2de08d">

## Steps to Review
1. Review the Test Plan for an overview of behavior
2. Review Integration tests for an overview of implementation and edge
cases
3. Review Cypress tests for an overview of UX changes
4. Testing on [Demo
Instance](https://rylnd-pr-181926-ml-rule-alert-suppression.kbndev.co/)
(elastic/changeme)
1. This instance has the relevant feature flag enabled, has some sample
auditbeat data, as well as the [anomalies archive
data](https://github.com/elastic/kibana/tree/main/x-pack/test/functional/es_archives/security_solution/anomalies)
for the purposes of exercising an ML rule against "real" anomalies
    1. There are a few example rules in the default space:
1. A simple [query
rule](https://rylnd-pr-181926-ml-rule-alert-suppression.kbndev.co/app/security/rules/id/f6f5960d-7e4b-40c1-ae15-501112822130)
against auditbeat data
1. An [ML
rule](https://rylnd-pr-181926-ml-rule-alert-suppression.kbndev.co/app/security/rules/id/9122669e-b2e1-41ce-af25-eeae15aa9ece)
with per-execution suppression on both `by_field_name` and
`by_field_value` (which ends up not actually suppressing anything)
1. An [ML
rule](https://rylnd-pr-181926-ml-rule-alert-suppression.kbndev.co/app/security/rules/id/0aabc280-00bd-42d4-82e6-65997c751797)
with per-execution suppression on `by_field_name` (which suppresses all
anomalies into a single alert)

## Related Issues
- This feature was temporarily blocked by
#183100, but those changes are
now in this PR.

## Checklist
- [x] Functional changes are hidden behind a feature flag. If not
hidden, the PR explains why these changes are being implemented in a
long-living feature branch.
- [x] Functional changes are covered with a test plan and automated
tests.
    * [Test Plan](elastic/security-team#9279)
- [x] Stability of new and changed tests is verified using the [Flaky
Test Runner](https://ci-stats.kibana.dev/trigger_flaky_test_runner) in
both ESS and Serverless. By default, use 200 runs for ESS and 200 runs
for Serverless.
* [ESS - Cypress x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6449)
* [Serverless - Cypress x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6450)
* [ESS - API x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6447)
* [Serverless - API x
200](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/6448)
- [ ] Comprehensive manual testing is done by two engineers: the PR
author and one of the PR reviewers. Changes are tested in both ESS and
Serverless.
- [ ] Mapping changes are accompanied by a technical design document. It
can be a GitHub issue or an RFC explaining the changes. The design
document is shared with and approved by the appropriate teams and
individual stakeholders.
- [ ] (OPTIONAL) OpenAPI specs changes include detailed descriptions and
examples of usage and are ready to be released on
https://docs.elastic.co/api-reference. NOTE: This is optional because at
the moment we don't have yet any OpenAPI specs that would be fully
"documented" and "GA-ready" for publishing on
https://docs.elastic.co/api-reference.
- [ ] Functional changes are communicated to the Docs team. A ticket is
opened in https://github.com/elastic/security-docs using the [Internal
documentation request (Elastic
employees)](https://github.com/elastic/security-docs/issues/new?assignees=&labels=&projects=&template=docs-request-internal.yaml&title=%5BRequest%5D+)
template. The following information is included: feature flags used,
target ESS version, planned timing for ESS and Serverless releases.

---------

Co-authored-by: Nastasha Solomon <79124755+nastasha-solomon@users.noreply.github.com>
Co-authored-by: kibanamachine <42973632+kibanamachine@users.noreply.github.com>

packages/kbn-es/src/serverless_resources/project_roles/security/roles.yml

@@ -35,6 +35,7 @@ viewer:
         - '.fleet-actions*'
         - 'risk-score.risk-score-*'
         - '.asset-criticality.asset-criticality-*'
+        - '.ml-anomalies-*'
       privileges:
         - read
   applications:
@@ -100,6 +101,10 @@ editor:
         - 'read'
         - 'write'
       allow_restricted_indices: false
+    - names:
+        - '.ml-anomalies-*'
+      privileges:
+        - read
   applications:
     - application: 'kibana-.kibana'
       privileges:
@@ -154,6 +159,7 @@ t1_analyst:
         - '.fleet-actions*'
         - risk-score.risk-score-*
         - .asset-criticality.asset-criticality-*
+        - '.ml-anomalies-*'
       privileges:
         - read
   applications:
@@ -201,6 +207,7 @@ t2_analyst:
         - .fleet-agents*
         - .fleet-actions*
         - risk-score.risk-score-*
+        - '.ml-anomalies-*'
       privileges:
         - read
     - names:
@@ -262,6 +269,7 @@ t3_analyst:
         - .fleet-agents*
         - .fleet-actions*
         - risk-score.risk-score-*
+        - '.ml-anomalies-*'
       privileges:
         - read
   applications:
@@ -331,6 +339,7 @@ threat_intelligence_analyst:
         - .fleet-agents*
         - .fleet-actions*
         - risk-score.risk-score-*
+        - '.ml-anomalies-*'
       privileges:
         - read
   applications:
@@ -389,6 +398,7 @@ rule_author:
         - .fleet-agents*
         - .fleet-actions*
         - risk-score.risk-score-*
+        - '.ml-anomalies-*'
       privileges:
         - read
   applications:
@@ -453,6 +463,7 @@ soc_manager:
         - .fleet-agents*
         - .fleet-actions*
         - risk-score.risk-score-*
+        - '.ml-anomalies-*'
       privileges:
         - read
   applications:
@@ -513,6 +524,7 @@ detections_admin:
         - metrics-endpoint.metadata_current_*
         - .fleet-agents*
         - .fleet-actions*
+        - '.ml-anomalies-*'
       privileges:
         - read
     - names:
@@ -570,6 +582,10 @@ platform_engineer:
       privileges:
         - read
         - write
+    - names:
+        - '.ml-anomalies-*'
+      privileges:
+        - read
   applications:
     - application: 'kibana-.kibana'
       privileges:
@@ -620,6 +636,7 @@ endpoint_operations_analyst:
         - .lists*
         - .items*
         - risk-score.risk-score-*
+        - '.ml-anomalies-*'
       privileges:
         - read
     - names:
@@ -710,6 +727,10 @@ endpoint_policy_manager:
         - read
         - write
         - manage
+    - names:
+        - '.ml-anomalies-*'
+      privileges:
+        - read
   applications:
     - application: 'kibana-.kibana'
       privileges:

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_request_schema.test.ts

@@ -1272,6 +1272,7 @@ describe('rules schema', () => {
       { ruleType: 'saved_query', ruleMock: getCreateSavedQueryRulesSchemaMock() },
       { ruleType: 'eql', ruleMock: getCreateEqlRuleSchemaMock() },
       { ruleType: 'new_terms', ruleMock: getCreateNewTermsRulesSchemaMock() },
+      { ruleType: 'machine_learning', ruleMock: getCreateMachineLearningRulesSchemaMock() },
     ];
 
     cases.forEach(({ ruleType, ruleMock }) => {

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.gen.ts

@@ -468,14 +468,25 @@ export const MachineLearningRuleRequiredFields = z.object({
   machine_learning_job_id: MachineLearningJobId,
 });
 
+export type MachineLearningRuleOptionalFields = z.infer<typeof MachineLearningRuleOptionalFields>;
+export const MachineLearningRuleOptionalFields = z.object({
+  alert_suppression: AlertSuppression.optional(),
+});
+
 export type MachineLearningRulePatchFields = z.infer<typeof MachineLearningRulePatchFields>;
-export const MachineLearningRulePatchFields = MachineLearningRuleRequiredFields.partial();
+export const MachineLearningRulePatchFields = MachineLearningRuleRequiredFields.partial().merge(
+  MachineLearningRuleOptionalFields
+);
 
 export type MachineLearningRuleResponseFields = z.infer<typeof MachineLearningRuleResponseFields>;
-export const MachineLearningRuleResponseFields = MachineLearningRuleRequiredFields;
+export const MachineLearningRuleResponseFields = MachineLearningRuleRequiredFields.merge(
+  MachineLearningRuleOptionalFields
+);
 
 export type MachineLearningRuleCreateFields = z.infer<typeof MachineLearningRuleCreateFields>;
-export const MachineLearningRuleCreateFields = MachineLearningRuleRequiredFields;
+export const MachineLearningRuleCreateFields = MachineLearningRuleRequiredFields.merge(
+  MachineLearningRuleOptionalFields
+);
 
 export type MachineLearningRule = z.infer<typeof MachineLearningRule>;
 export const MachineLearningRule = SharedResponseProps.merge(MachineLearningRuleResponseFields);

x-pack/plugins/security_solution/common/api/detection_engine/model/rule_schema/rule_schemas.schema.yaml

@@ -686,18 +686,27 @@ components:
         - machine_learning_job_id
         - anomaly_threshold
 
+    MachineLearningRuleOptionalFields:
+      type: object
+      properties:
+        alert_suppression:
+          $ref: './common_attributes.schema.yaml#/components/schemas/AlertSuppression'
+
     MachineLearningRulePatchFields:
       allOf:
         - $ref: '#/components/schemas/MachineLearningRuleRequiredFields'
           x-modify: partial
+        - $ref: '#/components/schemas/MachineLearningRuleOptionalFields'
 
     MachineLearningRuleResponseFields:
       allOf:
         - $ref: '#/components/schemas/MachineLearningRuleRequiredFields'
+        - $ref: '#/components/schemas/MachineLearningRuleOptionalFields'
 
     MachineLearningRuleCreateFields:
       allOf:
         - $ref: '#/components/schemas/MachineLearningRuleRequiredFields'
+        - $ref: '#/components/schemas/MachineLearningRuleOptionalFields'
 
     MachineLearningRule:
       allOf:

x-pack/plugins/security_solution/common/detection_engine/constants.ts

@@ -47,6 +47,7 @@ export const SUPPRESSIBLE_ALERT_RULES: Type[] = [
   'new_terms',
   'threat_match',
   'eql',
+  'machine_learning',
 ];
 
 export const SUPPRESSIBLE_ALERT_RULES_GA: Type[] = ['saved_query', 'query'];

x-pack/plugins/security_solution/common/detection_engine/utils.test.ts

@@ -236,9 +236,7 @@ describe('Alert Suppression Rules', () => {
       expect(isSuppressibleAlertRule('threat_match')).toBe(true);
       expect(isSuppressibleAlertRule('new_terms')).toBe(true);
       expect(isSuppressibleAlertRule('eql')).toBe(true);
-
-      // Rule types that don't support alert suppression:
-      expect(isSuppressibleAlertRule('machine_learning')).toBe(false);
+      expect(isSuppressibleAlertRule('machine_learning')).toBe(true);
     });
 
     test('should return false for an unknown rule type', () => {
@@ -273,9 +271,7 @@ describe('Alert Suppression Rules', () => {
       expect(isSuppressionRuleConfiguredWithDuration('threat_match')).toBe(true);
       expect(isSuppressionRuleConfiguredWithDuration('new_terms')).toBe(true);
       expect(isSuppressionRuleConfiguredWithDuration('eql')).toBe(true);
-
-      // Rule types that don't support alert suppression:
-      expect(isSuppressionRuleConfiguredWithDuration('machine_learning')).toBe(false);
+      expect(isSuppressionRuleConfiguredWithDuration('machine_learning')).toBe(true);
     });
 
     test('should return false for an unknown rule type', () => {
@@ -294,9 +290,7 @@ describe('Alert Suppression Rules', () => {
       expect(isSuppressionRuleConfiguredWithGroupBy('threat_match')).toBe(true);
       expect(isSuppressionRuleConfiguredWithGroupBy('new_terms')).toBe(true);
       expect(isSuppressionRuleConfiguredWithGroupBy('eql')).toBe(true);
-
-      // Rule types that don't support alert suppression:
-      expect(isSuppressionRuleConfiguredWithGroupBy('machine_learning')).toBe(false);
+      expect(isSuppressionRuleConfiguredWithGroupBy('machine_learning')).toBe(true);
     });
 
     test('should return false for a threshold rule type', () => {
@@ -320,9 +314,7 @@ describe('Alert Suppression Rules', () => {
       expect(isSuppressionRuleConfiguredWithMissingFields('threat_match')).toBe(true);
       expect(isSuppressionRuleConfiguredWithMissingFields('new_terms')).toBe(true);
       expect(isSuppressionRuleConfiguredWithMissingFields('eql')).toBe(true);
-
-      // Rule types that don't support alert suppression:
-      expect(isSuppressionRuleConfiguredWithMissingFields('machine_learning')).toBe(false);
+      expect(isSuppressionRuleConfiguredWithMissingFields('machine_learning')).toBe(true);
     });
 
     test('should return false for a threshold rule type', () => {

x-pack/plugins/security_solution/common/experimental_features.ts

@@ -175,6 +175,11 @@ export const allowedExperimentalValues = Object.freeze({
    */
   riskEnginePrivilegesRouteEnabled: true,
 
+  /**
+   * Enables alerts suppression for machine learning rules
+   */
+  alertSuppressionForMachineLearningRuleEnabled: false,
+
   /**
    * Enables experimental Experimental S1 integration data to be available in Analyzer
    */

x-pack/plugins/security_solution/public/common/components/ml/hooks/use_ml_rule_config.ts

@@ -0,0 +1,62 @@
+/*
+ * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one
+ * or more contributor license agreements. Licensed under the Elastic License
+ * 2.0; you may not use this file except in compliance with the Elastic License
+ * 2.0.
+ */
+
+import { useMemo } from 'react';
+import type { DataViewFieldBase } from '@kbn/es-query';
+
+import { getTermsAggregationFields } from '../../../../detection_engine/rule_creation_ui/components/step_define_rule/utils';
+import { useRuleFields } from '../../../../detection_engine/rule_management/logic/use_rule_fields';
+import type { BrowserField } from '../../../containers/source';
+import { useMlCapabilities } from './use_ml_capabilities';
+import { useMlRuleValidations } from './use_ml_rule_validations';
+import { hasMlAdminPermissions } from '../../../../../common/machine_learning/has_ml_admin_permissions';
+import { hasMlLicense } from '../../../../../common/machine_learning/has_ml_license';
+
+export interface UseMlRuleConfigReturn {
+  hasMlAdminPermissions: boolean;
+  hasMlLicense: boolean;
+  mlFields: DataViewFieldBase[];
+  mlFieldsLoading: boolean;
+  mlSuppressionFields: BrowserField[];
+  noMlJobsStarted: boolean;
+  someMlJobsStarted: boolean;
+}
+
+/**
+ * This hook is used to retrieve the various configurations and status needed for creating/editing an ML Rule in the Detection Engine UI. It composes several other ML hooks.
+ *
+ * @param machineLearningJobId The ID(s) of the ML job to retrieve the configuration for
+ *
+ * @returns {UseMlRuleConfigReturn} An object containing the various configurations and statuses needed for creating/editing an ML Rule
+ *
+ */
+export const useMLRuleConfig = ({
+  machineLearningJobId,
+}: {
+  machineLearningJobId: string[];
+}): UseMlRuleConfigReturn => {
+  const mlCapabilities = useMlCapabilities();
+  const { someJobsStarted: someMlJobsStarted, noJobsStarted: noMlJobsStarted } =
+    useMlRuleValidations({ machineLearningJobId });
+  const { loading: mlFieldsLoading, fields: mlFields } = useRuleFields({
+    machineLearningJobId,
+  });
+  const mlSuppressionFields = useMemo(
+    () => getTermsAggregationFields(mlFields as BrowserField[]),
+    [mlFields]
+  );
+
+  return {
+    hasMlAdminPermissions: hasMlAdminPermissions(mlCapabilities),
+    hasMlLicense: hasMlLicense(mlCapabilities),
+    mlFields,
+    mlFieldsLoading,
+    mlSuppressionFields,
+    noMlJobsStarted,
+    someMlJobsStarted,
+  };
+};