Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing #77040

Merged
merged 7 commits into from
Sep 15, 2020
Merged

[Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing #77040

merged 7 commits into from
Sep 15, 2020

Conversation

gmmorris
Copy link
Contributor

@gmmorris gmmorris commented Sep 9, 2020 •
edited
Loading

Summary

closes #76960

The Alerts Authorisation now manually builds the authorization filter instead of parsing a KQL string in order to get around the performance issue we recently identified in KQL. #76811

Initial testing showed a marked improvement, where the average duration of the find api in our integration test improved from an average of 386.14359232038ms (median of 199.40039098263ms) to an average of 108.44588107765ms (median of 89.378548502922ms).

Checklist

Delete any items that are not applicable to this PR.

For maintainers

@gmmorris gmmorris added Feature:Alerting Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.10.0 v8.0.0 labels Sep 9, 2020
@gmmorris gmmorris changed the title [Alerting] Improves pref of the authorization filter in AlertsClient.find by skipping KQL parsing [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing Sep 9, 2020
@gmmorris gmmorris marked this pull request as ready for review September 9, 2020 16:15
@gmmorris gmmorris requested a review from a team as a code owner September 9, 2020 16:15
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-alerting-services (Team:Alerting Services)

gmmorris and others added 2 commits September 10, 2020 08:24
@gmmorris @kobelb
optimized the is node
cb1fc7a 
Co-authored-by: Brandon Kobel <brandon.kobel@gmail.com>
@gmmorris
Merge branch 'master' into alerts/improve-find-pref
9d86fcd 
* master: (41 commits)
  [bugfix] Replace panel flyout opens 2 flyouts (elastic#76931)
  clean up test (elastic#76887)
  [Enterprise Search] Update shared API request handler (elastic#77112)
  [Maps] convert ESAggSource to TS (elastic#76999)
  Add plugin status API - take 2 (elastic#76732)
  Adds lens as a readable saved object for read-only dashboard users (elastic#77067)
  Skip checking for the reserved realm (elastic#76687)
  Reporting/diagnostics (elastic#74314)
  Reporting/Test: unskip non-screenshot tests (elastic#77088)
  Move metrics to setup and add cgroup metrics (elastic#76730)
  [Enterprise Search] Add Overview landing page/plugin (elastic#76734)
  First pass. Change TS type. Update OpenAPI (elastic#76434)
  [CI] Balance xpack ci groups a bit (elastic#77068)
  [Security_solution][Detections] Refactor signal ancestry to allow multiple parents (elastic#76531)
  [Maps] convert MetricsEditor to TS (elastic#76727)
  IndexMigrator: fix non blocking migration wrapper promise rejection (elastic#77018)
  [Enterprise Search] Update config data endpoint to v2 (elastic#76970)
  [ML] Add decision path charts to exploration results table (elastic#73561)
  Bump eventemitter3 from 4.0.0 to 4.0.7 (elastic#77016)
  [Ingest Pipelines] Add descriptions for ingest processors K-S (elastic#76981)
  ...
kobelb
kobelb approved these changes Sep 10, 2020
View reviewed changes
Copy link
Contributor

@kobelb kobelb left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The way we're building the KueryNode looks 👍 to me! I didn't thoroughly review the rest of the PR, so we'll want to get another approval from the Alerting team also.

@mikecote mikecote self-requested a review September 10, 2020 15:21
mikecote
mikecote approved these changes Sep 10, 2020
View reviewed changes
Copy link
Contributor

@mikecote mikecote left a comment •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Changes LGTM! Great to see an improvement in efficiency!

@gmmorris
Copy link
Contributor Author

@elasticmachine merge upstream

YulNaumenko
YulNaumenko approved these changes Sep 14, 2020
View reviewed changes
Copy link
Contributor

@YulNaumenko YulNaumenko left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@gmmorris
Merge branch 'master' into alerts/improve-find-pref
b4729a4 
* master: (65 commits)
  [Security Solution][Resolver] Analyzed event styling (elastic#77115)
  filter invalid SOs from the searc hresults in Task Manager (elastic#76891)
  [RUM Dashboard] Visitors by region map (elastic#77135)
  [Security Solution][Endpoint][Admin] Task/endpoint list actions (elastic#76555)
  [Ingest pipelines] Forms for processors T-U (elastic#76710)
  updating datatable type (elastic#77320)
  [ML] Fix custom URLs processing for security app (elastic#76957)
  [telemetry] add schema guideline + schema_check new check for --path config (elastic#75747)
  [ML] Transforms: API schemas and integration tests (elastic#75164)
  [Mappings editor] Add support for wildcard field type (elastic#76574)
  [Ingest Manager] Fix flyout instruction selection (elastic#77071)
  [Telemetry Tools] update lodash to 4.17 (elastic#77317)
  [APM] Service inventory redesign (elastic#76744)
  Hide management sections based on cluster/index privileges (elastic#67791)
  [Snapshot Restore] Disable steps when form is invalid (elastic#76540)
  [Mappings editor] Add support for positive_score_impact to rank_feature (elastic#76824)
  Update apm.ts (elastic#77310)
  [OBS] Remove beta badge, change news feed size and add external icon to news feed link (elastic#77164)
  [Discover] Convert legacy sort to be compatible with multi sort (elastic#76986)
  [APM] API Snapshot Testing (elastic#77229)
  ...
@gmmorris
Merge branch 'alerts/improve-find-pref' of github.com:gmmorris/kibana…
1e3fe6c 
… into alerts/improve-find-pref

* 'alerts/improve-find-pref' of github.com:gmmorris/kibana:
@kibanamachine
Copy link
Contributor

💚 Build Succeeded

Build metrics

distributable file count

id value diff baseline
default 45541 +2 45539

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@gmmorris gmmorris merged commit 223a187 into elastic:master Sep 15, 2020
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
@gmmorris
[Alerting] Improves performance of the authorization filter in Alerts…
af3cce2 
…Client.find by skipping KQL parsing (elastic#77040)

The Alerts Authorisation now manually builds the authorization filter instead of parsing a KQL string in order to get around the performance issue we recently identified in KQL.
@gmmorris gmmorris mentioned this pull request Sep 15, 2020
Merged
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
@gmmorris
Merge branch 'master' into alerting-exempt-rbac
696a7cb 
* master: (25 commits)
  [Security Solution] Add unit tests for Network search strategy (elastic#77416)
  [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040)
  [Ingest Manager] Add route for package installation by upload (elastic#77044)
  [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359)
  [APM] Remove useLocation and some minor route improvements (elastic#76343)
  [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258)
  [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444)
  Move remaining uses of serviceName away from urlParams (elastic#77248)
  [Lens] Move configuration popover to flyout (elastic#76046)
  [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589)
  skip flaky suite (elastic#59975)
  Neutral-naming in reporting plugin (elastic#77371)
  [Enterprise Search] Add UserIcon styles (elastic#77385)
  [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201)
  [Ingest Manager] Fix polling for new agent action (elastic#77339)
  Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212)
  Stacked headers and navigational search (elastic#72331)
  [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326)
  [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099)
  Add Lens to Recently Accessed (elastic#77249)
  ...
gmmorris added a commit to gmmorris/kibana that referenced this pull request Sep 15, 2020
@gmmorris
Merge branch 'master' into actions/webhook-remove-header
17a325d 
* master: (293 commits)
  Fix tsvb filter ration for table (elastic#77272)
  [Security Solution] Add unit tests for Network search strategy (elastic#77416)
  [Alerting] Improves performance of the authorization filter in AlertsClient.find by skipping KQL parsing (elastic#77040)
  [Ingest Manager] Add route for package installation by upload (elastic#77044)
  [APM-UI][E2E] filter PRs from the uptime GH team (elastic#77359)
  [APM] Remove useLocation and some minor route improvements (elastic#76343)
  [Enterprise Search] Update enterpriseSearchRequestHandler to manage range of errors + add handleAPIErrors helper (elastic#77258)
  [SECURITY_SOLUTION] Task/hostname policy response ux updates (elastic#76444)
  Move remaining uses of serviceName away from urlParams (elastic#77248)
  [Lens] Move configuration popover to flyout (elastic#76046)
  [Ingest Manager] Manually build Fleet kuery with Node arguments (elastic#76589)
  skip flaky suite (elastic#59975)
  Neutral-naming in reporting plugin (elastic#77371)
  [Enterprise Search] Add UserIcon styles (elastic#77385)
  [RUM Dashboard] Added loading state to visitor breakdown pie charts (elastic#77201)
  [Ingest Manager] Fix polling for new agent action (elastic#77339)
  Remote cluster - Functional UI test to change the superuser to a test_user with limited role (elastic#77212)
  Stacked headers and navigational search (elastic#72331)
  [ML] DF Analytics creation wizard: Fixing field loading race condition (elastic#77326)
  [Monitoring] Handle no mappings found for sort and collapse fields (elastic#77099)
  ...
gmmorris added a commit that referenced this pull request Sep 15, 2020
@gmmorris
[Alerting] Improves performance of the authorization filter in Alerts…
18d48aa 
…Client.find by skipping KQL parsing (#77040) (#77439)

The Alerts Authorisation now manually builds the authorization filter instead of parsing a KQL string in order to get around the performance issue we recently identified in KQL.
@dhurley14 dhurley14 mentioned this pull request Sep 21, 2020
Closed
2 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@YulNaumenko YulNaumenko YulNaumenko approved these changes

@mikecote mikecote mikecote approved these changes

@kobelb kobelb kobelb approved these changes

Assignees
No one assigned
Labels
Feature:Alerting release_note:enhancement Team:ResponseOps Label for the ResponseOps team (formerly the Cases and Alerting teams) v7.10.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Alerting RBAC - manually build KueryNode
6 participants
@gmmorris @elasticmachine @kibanamachine @kobelb @mikecote @YulNaumenko