Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[ML] Alerts-as-data for Anomaly detection rules #165958

Closed
darnautov opened this issue Sep 7, 2023 · 2 comments
Closed

[ML] Alerts-as-data for Anomaly detection rules #165958

darnautov opened this issue Sep 7, 2023 · 2 comments
Assignees
@darnautov
Labels
Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types Feature:Anomaly Detection ML anomaly detection :ml Team:ML Team label for ML (also use :ml) v8.11.0

Comments

@darnautov
Copy link
Contributor

With a persistent data layer available in the Kibana Alerting framework, it's possible to preserve alerting history for anomaly detection rules. We need to replace AlertFactory with AlertsClient so the alerts are persistent in an alert-as-data index.
@darnautov darnautov added :ml Feature:Anomaly Detection ML anomaly detection Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types Team:ML Team label for ML (also use :ml) Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry v8.11.0 labels Sep 7, 2023
@darnautov darnautov self-assigned this Sep 7, 2023
@elasticmachine
Copy link
Contributor

Pinging @elastic/ml-ui (:ml)

@darnautov darnautov mentioned this issue Sep 13, 2023
Merged
2 tasks
darnautov added a commit that referenced this issue Sep 28, 2023
@darnautov
[ML] Alerts as data integration for Anomaly Detection rule type (#166349
3ad5add 
)

## Summary

Part of #165958

Replaces usage of the deprecated `alertFactory` with the new alerts
client and adds alerts-as-data integration for Anomaly Detection
alerting rule type.

Alert instances are stored in
`.alerts-ml.anomaly-detection.alerts-default` index and extends the
common `AlertSchema`.

<details>
  <summary>Result mappings</summary>
  
  ```json
{
  ".internal.alerts-ml.anomaly-detection.alerts-default-000001": {
    "mappings": {
      "dynamic": "false",
      "_meta": {
        "namespace": "default",
        "kibana": {
          "version": "8.11.0"
        },
        "managed": true
      },
      "properties": {
        "@timestamp": {
          "type": "date"
        },
        "event": {
          "properties": {
            "action": {
              "type": "keyword"
            },
            "kind": {
              "type": "keyword"
            }
          }
        },
        "kibana": {
          "properties": {
            "alert": {
              "properties": {
                "action_group": {
                  "type": "keyword"
                },
                "anomaly_score": {
                  "type": "double"
                },
                "anomaly_timestamp": {
                  "type": "date"
                },
                "case_ids": {
                  "type": "keyword"
                },
                "duration": {
                  "properties": {
                    "us": {
                      "type": "long"
                    }
                  }
                },
                "end": {
                  "type": "date"
                },
                "flapping": {
                  "type": "boolean"
                },
                "flapping_history": {
                  "type": "boolean"
                },
                "instance": {
                  "properties": {
                    "id": {
                      "type": "keyword"
                    }
                  }
                },
                "is_interim": {
                  "type": "boolean"
                },
                "job_id": {
                  "type": "keyword"
                },
                "last_detected": {
                  "type": "date"
                },
                "maintenance_window_ids": {
                  "type": "keyword"
                },
                "reason": {
                  "type": "keyword"
                },
                "rule": {
                  "properties": {
                    "category": {
                      "type": "keyword"
                    },
                    "consumer": {
                      "type": "keyword"
                    },
                    "execution": {
                      "properties": {
                        "uuid": {
                          "type": "keyword"
                        }
                      }
                    },
                    "name": {
                      "type": "keyword"
                    },
                    "parameters": {
                      "type": "flattened",
                      "ignore_above": 4096
                    },
                    "producer": {
                      "type": "keyword"
                    },
                    "revision": {
                      "type": "long"
                    },
                    "rule_type_id": {
                      "type": "keyword"
                    },
                    "tags": {
                      "type": "keyword"
                    },
                    "uuid": {
                      "type": "keyword"
                    }
                  }
                },
                "start": {
                  "type": "date"
                },
                "status": {
                  "type": "keyword"
                },
                "time_range": {
                  "type": "date_range",
                  "format": "epoch_millis||strict_date_optional_time"
                },
                "top_influencers": {
                  "type": "nested",
                  "dynamic": "false",
                  "properties": {
                    "influencer_field_name": {
                      "type": "keyword"
                    },
                    "influencer_field_value": {
                      "type": "keyword"
                    },
                    "influencer_score": {
                      "type": "double"
                    },
                    "initial_influencer_score": {
                      "type": "double"
                    },
                    "is_interim": {
                      "type": "boolean"
                    },
                    "job_id": {
                      "type": "keyword"
                    },
                    "timestamp": {
                      "type": "date"
                    }
                  }
                },
                "top_records": {
                  "type": "nested",
                  "dynamic": "false",
                  "properties": {
                    "actual": {
                      "type": "double"
                    },
                    "by_field_name": {
                      "type": "keyword"
                    },
                    "by_field_value": {
                      "type": "keyword"
                    },
                    "detector_index": {
                      "type": "integer"
                    },
                    "field_name": {
                      "type": "keyword"
                    },
                    "function": {
                      "type": "keyword"
                    },
                    "initial_record_score": {
                      "type": "double"
                    },
                    "is_interim": {
                      "type": "boolean"
                    },
                    "job_id": {
                      "type": "keyword"
                    },
                    "over_field_name": {
                      "type": "keyword"
                    },
                    "over_field_value": {
                      "type": "keyword"
                    },
                    "partition_field_name": {
                      "type": "keyword"
                    },
                    "partition_field_value": {
                      "type": "keyword"
                    },
                    "record_score": {
                      "type": "double"
                    },
                    "timestamp": {
                      "type": "date"
                    },
                    "typical": {
                      "type": "double"
                    }
                  }
                },
                "url": {
                  "type": "keyword",
                  "index": false,
                  "ignore_above": 2048
                },
                "uuid": {
                  "type": "keyword"
                },
                "workflow_status": {
                  "type": "keyword"
                },
                "workflow_tags": {
                  "type": "keyword"
                }
              }
            },
            "space_ids": {
              "type": "keyword"
            },
            "version": {
              "type": "version"
            }
          }
        },
        "tags": {
          "type": "keyword"
        }
      }
    }
  }
}
  ```
</details>

### Checklist

- [ ]
[Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html)
was added for features that require explanation or tutorials
- [x] [Unit or functional
tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html)
were updated or added to match the most common scenarios
@peteharverson
Copy link
Contributor

Closing as this work has been completed for 8.11 - see #166349.

This was referenced Oct 13, 2023
Merged
Merged
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@darnautov darnautov

Labels
Feature:Alerting/Alerts-as-Data Issues related to Alerts-as-data and RuleRegistry Feature:Alerting/RuleTypes Issues related to specific Alerting Rules Types Feature:Anomaly Detection ML anomaly detection :ml Team:ML Team label for ML (also use :ml) v8.11.0
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
3 participants
@darnautov @peteharverson @elasticmachine