[SIEM][Detection Engine] Create REST API endpoint for KQL signals #47013
Assignees
Labels
Comments
|
Pinging @elastic/siem |
FrankHassanabad
added a commit
that referenced
this issue
Oct 7, 2019
## Summary * #47013 This adds a basic create signal API rout API at: ``` /api/siem/signals ``` At the moment this will create an action/alert as a one for one. This also introduces several scripts which allow developers to: * Re-create signals index for development velocity * post signals to the `/api/siem/signals` endpoint * Check different indexes, saved objects, and task manager for trouble shooting ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ ~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this issue
Oct 7, 2019
## Summary * elastic#47013 This adds a basic create signal API rout API at: ``` /api/siem/signals ``` At the moment this will create an action/alert as a one for one. This also introduces several scripts which allow developers to: * Re-create signals index for development velocity * post signals to the `/api/siem/signals` endpoint * Check different indexes, saved objects, and task manager for trouble shooting ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ ~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
FrankHassanabad
added a commit
that referenced
this issue
Oct 7, 2019
## Summary * #47013 This adds a basic create signal API rout API at: ``` /api/siem/signals ``` At the moment this will create an action/alert as a one for one. This also introduces several scripts which allow developers to: * Re-create signals index for development velocity * post signals to the `/api/siem/signals` endpoint * Check different indexes, saved objects, and task manager for trouble shooting ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ ~~- [ ] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios~~ ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
This was referenced Oct 15, 2019
FrankHassanabad
added a commit
that referenced
this issue
Oct 17, 2019
…r product testing, and more unit tests (#48471) ## Summary * Adds stable alerting id's by using the alert params. * Currently does a manual walk through of the alert params to find the stable id * Updated all of the endpoints to take either of the two id's. * Added several scripts to support performance testing ad-hoc such as `post_x_signals.sh` * Added scripts to support converting from saved searches to alerts. * Consolidated and fixed a lot of the backend types * Added unit tests against the router endpoints * #47013 ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) - [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this issue
Oct 17, 2019
…r product testing, and more unit tests (elastic#48471) * Adds stable alerting id's by using the alert params. * Currently does a manual walk through of the alert params to find the stable id * Updated all of the endpoints to take either of the two id's. * Added several scripts to support performance testing ad-hoc such as `post_x_signals.sh` * Added scripts to support converting from saved searches to alerts. * Consolidated and fixed a lot of the backend types * Added unit tests against the router endpoints * elastic#47013 Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) - [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
FrankHassanabad
added a commit
that referenced
this issue
Oct 18, 2019
…r product testing, and more unit tests (#48471) (#48580) * Adds stable alerting id's by using the alert params. * Currently does a manual walk through of the alert params to find the stable id * Updated all of the endpoints to take either of the two id's. * Added several scripts to support performance testing ad-hoc such as `post_x_signals.sh` * Added scripts to support converting from saved searches to alerts. * Consolidated and fixed a lot of the backend types * Added unit tests against the router endpoints * #47013 Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ - [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process) - [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)
1 task
FrankHassanabad
added a commit
that referenced
this issue
Oct 28, 2019
#49451) ## Summary * Removes the older beginner KQL type of signal creation in favor of newer version with filtering * Adds ability to create KQL or lucene queries that will work with the UI filters * UI state with the filters are now savable to re-hydrate UI's on the front end * Adds `saved_id` ability so the UI can tether dynamic saved queries with signals * Changed `it` to `test` as `it` is not the alias we use for tests * Updated script which converts older saved searches to work with newer mechanism * Fixed script to accept proper ndjson lines * Adds validation unit tests for the endpoint * Increases validation strictness of the endpoints * Adds more data scripts for testing scenarios * #47013 ## Testing * Run `./hard_reset.sh` script * Test with both algorithms through this toggle before starting kibana: `export USE_REINDEX_API=true` * Convert older saved searches to compatible new query filters by running: `./convert_saved_search_to_signals.sh ~/projects/saved_searches /tmp/signals` * Post them`./post_signal.sh /tmp/signals/*.json` * Hard reset again * Test smaller set of signals and REST endpoints using the typical scripts of: ```sh ./post_signal.sh ./read_signal.sh ./find_signals.sh ./update_signal.sh ./delete_signal.sh ``` or test using POSTMAN, etc... If you want to test validation. If you see any validation issues let me know as I have validation testing files and can easily fix them add another unit test to the growing large collection we have now. Change in your advanced settings of SIEM to use your signals index you configured for verification that the signals show up. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
FrankHassanabad
added a commit
to FrankHassanabad/kibana
that referenced
this issue
Oct 28, 2019
elastic#49451) ## Summary * Removes the older beginner KQL type of signal creation in favor of newer version with filtering * Adds ability to create KQL or lucene queries that will work with the UI filters * UI state with the filters are now savable to re-hydrate UI's on the front end * Adds `saved_id` ability so the UI can tether dynamic saved queries with signals * Changed `it` to `test` as `it` is not the alias we use for tests * Updated script which converts older saved searches to work with newer mechanism * Fixed script to accept proper ndjson lines * Adds validation unit tests for the endpoint * Increases validation strictness of the endpoints * Adds more data scripts for testing scenarios * elastic#47013 ## Testing * Run `./hard_reset.sh` script * Test with both algorithms through this toggle before starting kibana: `export USE_REINDEX_API=true` * Convert older saved searches to compatible new query filters by running: `./convert_saved_search_to_signals.sh ~/projects/saved_searches /tmp/signals` * Post them`./post_signal.sh /tmp/signals/*.json` * Hard reset again * Test smaller set of signals and REST endpoints using the typical scripts of: ```sh ./post_signal.sh ./read_signal.sh ./find_signals.sh ./update_signal.sh ./delete_signal.sh ``` or test using POSTMAN, etc... If you want to test validation. If you see any validation issues let me know as I have validation testing files and can easily fix them add another unit test to the growing large collection we have now. Change in your advanced settings of SIEM to use your signals index you configured for verification that the signals show up. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
FrankHassanabad
added a commit
that referenced
this issue
Oct 29, 2019
#49451) (#49583) ## Summary * Removes the older beginner KQL type of signal creation in favor of newer version with filtering * Adds ability to create KQL or lucene queries that will work with the UI filters * UI state with the filters are now savable to re-hydrate UI's on the front end * Adds `saved_id` ability so the UI can tether dynamic saved queries with signals * Changed `it` to `test` as `it` is not the alias we use for tests * Updated script which converts older saved searches to work with newer mechanism * Fixed script to accept proper ndjson lines * Adds validation unit tests for the endpoint * Increases validation strictness of the endpoints * Adds more data scripts for testing scenarios * #47013 ## Testing * Run `./hard_reset.sh` script * Test with both algorithms through this toggle before starting kibana: `export USE_REINDEX_API=true` * Convert older saved searches to compatible new query filters by running: `./convert_saved_search_to_signals.sh ~/projects/saved_searches /tmp/signals` * Post them`./post_signal.sh /tmp/signals/*.json` * Hard reset again * Test smaller set of signals and REST endpoints using the typical scripts of: ```sh ./post_signal.sh ./read_signal.sh ./find_signals.sh ./update_signal.sh ./delete_signal.sh ``` or test using POSTMAN, etc... If you want to test validation. If you see any validation issues let me know as I have validation testing files and can easily fix them add another unit test to the growing large collection we have now. Change in your advanced settings of SIEM to use your signals index you configured for verification that the signals show up. ### Checklist Use ~~strikethroughs~~ to remove checklist items you don't feel are applicable to this PR. ~~- [ ] This was checked for cross-browser compatibility, [including a check against IE11](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility)~~ ~~- [ ] Any text added follows [EUI's writing guidelines](https://elastic.github.io/eui/#/guidelines/writing), uses sentence case text and includes [i18n support](https://github.com/elastic/kibana/blob/master/packages/kbn-i18n/README.md)~~ ~~- [ ] [Documentation](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#writing-documentation) was added for features that require explanation or tutorials~~ - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios ~~- [ ] This was checked for [keyboard-only and screenreader accessibility](https://developer.mozilla.org/en-US/docs/Learn/Tools_and_testing/Cross_browser_testing/Accessibility#Accessibility_testing_checklist)~~ ### For maintainers ~~- [ ] This was checked for breaking API changes and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~ ~~- [ ] This includes a feature addition or change that requires a release note and was [labeled appropriately](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#release-notes-process)~~
20 tasks
|
We will have additional REST API endpoints until the heat death of the universe I am sure. This will be closed now as we have the majority of the endpoints completed. Individual requests can be new tickets. |
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Create the REST API endpoints for the Kibana UI to consume with regards to the KQL signals.
These API's should allow the the common abstractions expected for a KQL signal which is:
The text was updated successfully, but these errors were encountered: