Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' #79056

YulNaumenko · 2020-10-01T06:12:42Z

Current PR include the next features:

Exposed AlertsClient method getHealth, which is using executionStatus alert property to verify if any of alerts in the system has an errors of specific reason.
Added a task 'alerting_health_check' for checking framework decryption failures, which is executing for every hour and using AlertsClient method getHealth().hasDecryptionFailures.
Extended alerts plugin setup with updating core status, by using API core.status.set. Ne core status get the latest health state from 'alerting_health_check' task execution result.
Extended api/alerts/_health with the new property alertingFrameworkHeath.

…tatus pooler for Kibana status plugins 'kibanahost/api/status'

elasticmachine · 2020-10-01T06:12:44Z

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

…k-health-status # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

…ed correct health task implementation

…k-health-status # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

…heck

x-pack/plugins/alerts/server/alerts_client.ts

pmuellr

I left a bunch of comments, which suggest some different shape changes on things - which may not be right, not sure :-). Perhaps would be worth a call to go over, I may be misunderstanding the design.

x-pack/plugins/alerts/server/alerts_client.ts

x-pack/plugins/alerts/server/health/get_state.ts

x-pack/plugins/alerts/common/index.ts

x-pack/plugins/alerts/server/plugin.ts

…k-health-status # Please enter a commit message to explain why this merge is necessary, # especially if it merges an updated upstream into a topic branch. # # Lines starting with '#' will be ignored, and an empty message aborts # the commit.

dhurley14 · 2020-10-29T23:21:41Z

x-pack/plugins/security_solution/server/lib/detection_engine/routes/utils.test.ts

@@ -464,7 +465,7 @@ describe('utils', () => {
        status: 'error',
        lastExecutionDate: foundRule.executionStatus.lastExecutionDate,
        error: {
-          reason: 'read',
+          reason: AlertExecutionStatusErrorReasons.Read,


Thanks for the cleanup here and above at https://github.com/elastic/kibana/pull/79056/files#diff-f6140d9a8de26e26d80f53651169d01af3446e59c854324b05f492fb708405cbR61 !

…k-health-status # Conflicts: # x-pack/plugins/triggers_actions_ui/public/application/sections/alerts_list/components/alerts_list.test.tsx

pmuellr

LGTM, left a few comments/questions

x-pack/plugins/alerts/server/alerts_client/alerts_client.ts

pmuellr · 2020-11-02T22:17:22Z

x-pack/plugins/alerts/server/health/task.ts

+) {
+  try {
+    const interval = (await config).healthCheck.interval;
+    await taskManager.ensureScheduled({


I was anxious to see this running, so hacked an alert executor to throw an error, and then the config to run every minute :-). Makes me wonder if we want to schedule an additional one-time call at startup (maybe 1 minute after startup), so we can get the latest data shortly after startup, without having to wait for the hourly task to run. (could be another issue/PR)

gmmorris

We're getting much closer :)

There are a few things that still bug me (some duplication of code and nit. stuff) but the thing that is still keeping me from approving is that I don't understand why getHealth is on the AlertsClient instead of the Plugin Contract.

It doesn't use any user specific privileges, requires a fake request to be used in the task, and feels like a Plugin level API, so I think it should be moved there...

x-pack/plugins/alerts/server/health/task.ts

x-pack/plugins/alerts/server/health/get_state.ts

x-pack/plugins/alerts/server/health/task.ts

x-pack/plugins/alerts/server/health/get_state.ts

x-pack/plugins/alerts/server/alerts_client/alerts_client.ts

x-pack/plugins/alerts/server/health/get_state.ts

x-pack/plugins/alerts/server/health/task.ts

x-pack/plugins/alerts/server/health/get_state.ts

x-pack/plugins/alerts/server/plugin.ts

…k-health-status # Conflicts: # x-pack/plugins/alerts/server/alerts_client/alerts_client.ts

…k-health-status

x-pack/plugins/alerts/server/health/get_health.ts

x-pack/plugins/alerts/server/plugin.ts

gmmorris

LGTM!
I think some dead code has slipped through in plugin.ts, but other than that this is looking great! Well done :)

kibanamachine · 2020-11-06T23:31:19Z

💚 Build Succeeded

continuous-integration/kibana-ci/pull-request
Commit: bc88bdf

Metrics [docs]

distributable file count

id	before	after	diff
`default`	42731	42736	+5

page load bundle size

id	before	after	diff
`alerts`	87.8KB	88.4KB	+665.0B

History

💔 Build #86511 failed e23eb26
💛 Build #86490 was flaky 169e946
💔 Build #86319 failed 75d7eea
💚 Build #85732 succeeded b344aeb
💔 Build #85730 failed 397a622

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

…tatus pooler for Kibana status plugins 'kibanahost/api/status' (elastic#79056) * Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' * Exposed health task registration to alerts plugin * Fixed type error * Extended health API endpoint with info about decryption failures, added correct health task implementation * adjusted query * Tested locally and got it working as expected, fixed tests and type check * Added unit tests * Changed AlertExecutionStatusErrorReasons to be enum * Uppercase the enum * Replaced string values to enum * Fixed types * Extended AlertsClient with getHealth method * added return type to healthStatus$ * Added configurable health check interval and timestamps * Extended update core status interval to 5mins * Fixed failing tests * Registered alerts config * Fixed date for ok health state * fixed jest test * fixed task state * Fixed due to comments, moved getHealth to a plugin level * fixed type checks * Added sorting to the latest Ok state last update * adjusted error queries * Fixed jest tests * removed unused * fixed type check

…tatus pooler for Kibana status plugins 'kibanahost/api/status' (#79056) (#82907) * Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' * Exposed health task registration to alerts plugin * Fixed type error * Extended health API endpoint with info about decryption failures, added correct health task implementation * adjusted query * Tested locally and got it working as expected, fixed tests and type check * Added unit tests * Changed AlertExecutionStatusErrorReasons to be enum * Uppercase the enum * Replaced string values to enum * Fixed types * Extended AlertsClient with getHealth method * added return type to healthStatus$ * Added configurable health check interval and timestamps * Extended update core status interval to 5mins * Fixed failing tests * Registered alerts config * Fixed date for ok health state * fixed jest test * fixed task state * Fixed due to comments, moved getHealth to a plugin level * fixed type checks * Added sorting to the latest Ok state last update * adjusted error queries * Fixed jest tests * removed unused * fixed type check

* master: (68 commits) [Fleet] Make stream id unique in agent policy (elastic#82447) skip flaky suite (elastic#82915) skip flaky suite (elastic#75794) Copy `dateAsStringRt` to observability plugin (elastic#82839) [Maps] rename connected_components/map folder to mb_map (elastic#82897) [Security Solution] Fix EventsViewer DnD cypress tests (elastic#82619) [Security Solution] Adds logging and performance fan out API for threat/Indicator matching (elastic#82546) Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' (elastic#79056) [APM] Adds new configuration 'xpack.apm.maxServiceEnvironments' (elastic#82090) Move single use function in line (elastic#82885) [ML] Add unsigned_long support to data frame analytics and anomaly detection (elastic#82636) Add flot_chart dependency from shared_deps to Shareable Runtime (elastic#81649) [Security Solution][Detections] - Auto refresh all rules/monitoring tables (elastic#82062) [APM] Fix apm e2e runner script commands (elastic#82798) [Ingest Manager] Move cache functions to from registry to archive (elastic#82871) Update webpack-dev-server and webpack-cli (elastic#82844) [Uptime] Migrate to new es client (elastic#82003) Move parseAndVerify* functions to validation.ts (elastic#82845) Remove yeoman & yo (elastic#82825) [Canvas] Fix elements not being updated properly when filter is changed on workpad (elastic#81863) ...

…e-details-overlay * 'master' of github.com:elastic/kibana: (201 commits) Added `defaultActionMessage` to index threshold alert UI type definition (elastic#80936) [ILM] Migrate Delete phase and name field to Form Lib (elastic#82834) skip flaky suite (elastic#57426) [Alerting] adds an Run When field in the alert flyout to assign the action to an Action Group (elastic#82472) [APM] Expose APM event client as part of plugin contract (elastic#82724) [Logs UI] Fix errors during navigation (elastic#78319) Enable send to background in TSVB (elastic#82835) SavedObjects search_dsl: add match_phrase_prefix clauses when using prefix search (elastic#82693) [Ingest Manager] Unify install* under installPackage (elastic#82916) [Fleet] Make stream id unique in agent policy (elastic#82447) skip flaky suite (elastic#82915) skip flaky suite (elastic#75794) Copy `dateAsStringRt` to observability plugin (elastic#82839) [Maps] rename connected_components/map folder to mb_map (elastic#82897) [Security Solution] Fix EventsViewer DnD cypress tests (elastic#82619) [Security Solution] Adds logging and performance fan out API for threat/Indicator matching (elastic#82546) Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' (elastic#79056) [APM] Adds new configuration 'xpack.apm.maxServiceEnvironments' (elastic#82090) Move single use function in line (elastic#82885) [ML] Add unsigned_long support to data frame analytics and anomaly detection (elastic#82636) ...

Implemented Alerting health status pusher by using task manager and s…

27149da

…tatus pooler for Kibana status plugins 'kibanahost/api/status'

YulNaumenko added Feature:Alerting v8.0.0 release_note:skip Skip the PR/issue when compiling release notes Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.10.0 labels Oct 1, 2020

YulNaumenko requested a review from a team as a code owner October 1, 2020 06:12

YulNaumenko self-assigned this Oct 1, 2020

YulNaumenko added 10 commits October 1, 2020 14:25

Exposed health task registration to alerts plugin

36346a0

Fixed type error

ef4c938

Extended health API endpoint with info about decryption failures, add…

ff59fce

…ed correct health task implementation

adjusted query

c7d8b14

Tested locally and got it working as expected, fixed tests and type c…

c0d96b3

…heck

Added unit tests

3f52662

YulNaumenko requested a review from joshdover October 7, 2020 23:26

gmmorris self-requested a review October 8, 2020 10:17

gmmorris reviewed Oct 8, 2020

View reviewed changes

x-pack/plugins/alerts/server/alerts_client.ts Outdated Show resolved Hide resolved

pmuellr reviewed Oct 8, 2020

View reviewed changes

pmuellr added v7.11.0 and removed v7.10.0 labels Oct 8, 2020

YulNaumenko added 5 commits October 8, 2020 13:02

Changed AlertExecutionStatusErrorReasons to be enum

35eba5e

Uppercase the enum

f20a0e0

Replaced string values to enum

bde12d8

Fixed types

ea01e31

dhurley14 reviewed Oct 29, 2020

View reviewed changes

YulNaumenko added 2 commits October 30, 2020 17:20

fixed task state

0aeaf5d

Merge remote-tracking branch 'upstream/master' into alerting-framewor…

c3ae044

…k-health-status # Conflicts: # x-pack/plugins/triggers_actions_ui/public/application/sections/alerts_list/components/alerts_list.test.tsx

pmuellr approved these changes Nov 2, 2020

View reviewed changes

gmmorris suggested changes Nov 3, 2020

View reviewed changes

YulNaumenko added 2 commits November 3, 2020 17:14

Merge remote-tracking branch 'upstream/master' into alerting-framewor…

41e338a

…k-health-status # Conflicts: # x-pack/plugins/alerts/server/alerts_client/alerts_client.ts

Fixed due to comments, moved getHealth to a plugin level

397a622

YulNaumenko requested a review from gmmorris November 4, 2020 06:13

YulNaumenko added 4 commits November 3, 2020 22:36

fixed type checks

b344aeb

Merge remote-tracking branch 'upstream/master' into alerting-framewor…

5197255

…k-health-status

Added sorting to the latest Ok state last update

8964d24

adjusted error queries

75d7eea

gmmorris reviewed Nov 6, 2020

View reviewed changes

x-pack/plugins/alerts/server/health/get_health.ts Show resolved Hide resolved

gmmorris reviewed Nov 6, 2020

View reviewed changes

x-pack/plugins/alerts/server/plugin.ts Outdated Show resolved Hide resolved

gmmorris approved these changes Nov 6, 2020

View reviewed changes

YulNaumenko added 3 commits November 6, 2020 09:45

Fixed jest tests

169e946

removed unused

e23eb26

fixed type check

bc88bdf

YulNaumenko merged commit 802c6dc into elastic:master Nov 7, 2020

YulNaumenko mentioned this pull request Nov 7, 2020

[7.x] Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' (#79056) #82907

Merged

pmuellr mentioned this pull request Nov 19, 2020

[Actions] Add action group for notifying users of alert execution failure #83748

Closed

YulNaumenko added the needs_docs label Nov 19, 2020

ymao1 mentioned this pull request Dec 7, 2020

[Alerting & Actions ] More debug logging #85149

Merged

This was referenced Feb 25, 2021

Catch-up release documentation - @YulNaumenko #90008

Closed

[Alerts][Doc] Added README documentation for alerts plugin status and framework health checks configuration options. #92761

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' #79056

Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' #79056

YulNaumenko commented Oct 1, 2020 •

edited

Loading

elasticmachine commented Oct 1, 2020

pmuellr left a comment

dhurley14 Oct 29, 2020

pmuellr left a comment

pmuellr Nov 2, 2020

gmmorris left a comment

gmmorris left a comment

kibanamachine commented Nov 6, 2020

Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' #79056

Implemented Alerting health status pusher by using task manager and status pooler for Kibana status plugins 'kibanahost/api/status' #79056

Conversation

YulNaumenko commented Oct 1, 2020 • edited Loading

elasticmachine commented Oct 1, 2020

pmuellr left a comment

Choose a reason for hiding this comment

dhurley14 Oct 29, 2020

Choose a reason for hiding this comment

pmuellr left a comment

Choose a reason for hiding this comment

pmuellr Nov 2, 2020

Choose a reason for hiding this comment

gmmorris left a comment

Choose a reason for hiding this comment

gmmorris left a comment

Choose a reason for hiding this comment

kibanamachine commented Nov 6, 2020

💚 Build Succeeded

Metrics [docs]

distributable file count

page load bundle size

History

YulNaumenko commented Oct 1, 2020 •

edited

Loading