-
Notifications
You must be signed in to change notification settings - Fork 8.3k
Commit
This commit does not belong to any branch on this repository, and may belong to a fork outside of the repository.
[SIEM][Detection Engine][Lists] Adds the ability for exception lists …
…to be multi-list queried. (#71540) (#71580) ## Summary * Adds the ability for exception lists to be multi-list queried * Fixes a bunch of script issues where I did not update everywhere I needed to use `ip_list` and deletes an old list that now lives within the new/lists folder * Fixes a few io-ts issues with Encode Decode while I was in there. * Adds two more types and their tests for supporting converting between comma separated strings and arrays for GET calls. * Fixes one weird circular dep issue while adding more types. You now send into the find an optional comma separated list of exception lists their namespace type and any filters like so: ```ts GET /api/exception_lists/items/_find?list_id=simple_list,endpoint_list&namespace_type=single,agnostic&filtering=filter1,filter2" ``` And this will return the results of both together with each filter applied to each list. If you use a sort field and ordering it will order across the lists together as if they are one list. Filter is optional like before. If you provide less filters than there are lists, the lists will only apply the filters to each list until it runs out of filters and then not filter the other lists. If at least one list is found this will _not_ return a 404 but it will _only_ query the list(s) it did find. If none of the lists are found, then this will return a 404 not found exception. **Script testing** See these files for more information: * find_exception_list_items.sh * find_exception_list_items_by_filter.sh But basically you can create two lists and an item for each of the lists: ```ts ./post_exception_list.sh ./exception_lists/new/exception_list.json ./post_exception_list_item.sh ./exception_lists/new/exception_list_item.json ./post_exception_list.sh ./exception_lists/new/exception_list_agnostic.json ./post_exception_list_item.sh ./exception_lists/new/exception_list_item_agnostic.json ``` And then you can query these two lists together: ```ts ./find_exception_list_items.sh simple_list,endpoint_list single,agnostic ``` Or for filtering you can query both and add a filter for each one: ```ts ./find_exception_list_items_by_filter.sh simple_list,endpoint_list "exception-list.attributes.name:%20Sample%20Endpoint%20Exception%20List,exception-list-agnostic.attributes.name:%20Sample%20Endpoint%20Exception%20List" single,agnostic ``` ### Checklist Delete any items that are not applicable to this PR. - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios
- Loading branch information
1 parent
0dad095
commit 5b2d0b3
Showing
42 changed files
with
786 additions
and
143 deletions.
There are no files selected for viewing
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
99 changes: 99 additions & 0 deletions
99
x-pack/plugins/lists/common/schemas/types/default_namespace_array.test.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,99 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License; | ||
| * you may not use this file except in compliance with the Elastic License. | ||
| */ | ||
|
|
||
| import { pipe } from 'fp-ts/lib/pipeable'; | ||
| import { left } from 'fp-ts/lib/Either'; | ||
|
|
||
| import { foldLeftRight, getPaths } from '../../siem_common_deps'; | ||
|
|
||
| import { DefaultNamespaceArray, DefaultNamespaceArrayTypeEncoded } from './default_namespace_array'; | ||
|
|
||
| describe('default_namespace_array', () => { | ||
| test('it should validate "null" single item as an array with a "single" value', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = null; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual(['single']); | ||
| }); | ||
|
|
||
| test('it should NOT validate a numeric value', () => { | ||
| const payload = 5; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([ | ||
| 'Invalid value "5" supplied to "DefaultNamespaceArray"', | ||
| ]); | ||
| expect(message.schema).toEqual({}); | ||
| }); | ||
|
|
||
| test('it should validate "undefined" item as an array with a "single" value', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = undefined; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual(['single']); | ||
| }); | ||
|
|
||
| test('it should validate "single" as an array of a "single" value', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = 'single'; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual([payload]); | ||
| }); | ||
|
|
||
| test('it should validate "agnostic" as an array of a "agnostic" value', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = 'agnostic'; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual([payload]); | ||
| }); | ||
|
|
||
| test('it should validate "single,agnostic" as an array of 2 values of ["single", "agnostic"] values', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = 'agnostic,single'; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual(['agnostic', 'single']); | ||
| }); | ||
|
|
||
| test('it should validate 3 elements of "single,agnostic,single" as an array of 3 values of ["single", "agnostic", "single"] values', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = 'single,agnostic,single'; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual(['single', 'agnostic', 'single']); | ||
| }); | ||
|
|
||
| test('it should validate 3 elements of "single,agnostic, single" as an array of 3 values of ["single", "agnostic", "single"] values when there are spaces', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = ' single, agnostic, single '; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([]); | ||
| expect(message.schema).toEqual(['single', 'agnostic', 'single']); | ||
| }); | ||
|
|
||
| test('it should not validate 3 elements of "single,agnostic,junk" since the 3rd value is junk', () => { | ||
| const payload: DefaultNamespaceArrayTypeEncoded = 'single,agnostic,junk'; | ||
| const decoded = DefaultNamespaceArray.decode(payload); | ||
| const message = pipe(decoded, foldLeftRight); | ||
|
|
||
| expect(getPaths(left(message.errors))).toEqual([ | ||
| 'Invalid value "junk" supplied to "DefaultNamespaceArray"', | ||
| ]); | ||
| expect(message.schema).toEqual({}); | ||
| }); | ||
| }); |
45 changes: 45 additions & 0 deletions
45
x-pack/plugins/lists/common/schemas/types/default_namespace_array.ts
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,45 @@ | ||
| /* | ||
| * Copyright Elasticsearch B.V. and/or licensed to Elasticsearch B.V. under one | ||
| * or more contributor license agreements. Licensed under the Elastic License; | ||
| * you may not use this file except in compliance with the Elastic License. | ||
| */ | ||
|
|
||
| import * as t from 'io-ts'; | ||
| import { Either } from 'fp-ts/lib/Either'; | ||
|
|
||
| import { namespaceType } from './default_namespace'; | ||
|
|
||
| export const namespaceTypeArray = t.array(namespaceType); | ||
| export type NamespaceTypeArray = t.TypeOf<typeof namespaceTypeArray>; | ||
|
|
||
| /** | ||
| * Types the DefaultNamespaceArray as: | ||
| * - If null or undefined, then a default string array of "single" will be used. | ||
| * - If it contains a string, then it is split along the commas and puts them into an array and validates it | ||
| */ | ||
| export const DefaultNamespaceArray = new t.Type< | ||
| NamespaceTypeArray, | ||
| string | undefined | null, | ||
| unknown | ||
| >( | ||
| 'DefaultNamespaceArray', | ||
| namespaceTypeArray.is, | ||
| (input, context): Either<t.Errors, NamespaceTypeArray> => { | ||
| if (input == null) { | ||
| return t.success(['single']); | ||
| } else if (typeof input === 'string') { | ||
| const commaSeparatedValues = input | ||
| .trim() | ||
| .split(',') | ||
| .map((value) => value.trim()); | ||
| return namespaceTypeArray.validate(commaSeparatedValues, context); | ||
| } | ||
| return t.failure(input, context); | ||
| }, | ||
| String | ||
| ); | ||
|
|
||
| export type DefaultNamespaceC = typeof DefaultNamespaceArray; | ||
|
|
||
| export type DefaultNamespaceArrayTypeEncoded = t.OutputOf<typeof DefaultNamespaceArray>; | ||
| export type DefaultNamespaceArrayTypeDecoded = t.TypeOf<typeof DefaultNamespaceArray>; |
Oops, something went wrong.