[SIEM][Detection Engine][Lists] Adds version and immutability data structures #72730

FrankHassanabad · 2020-07-21T19:53:11Z

Summary

The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures.

Adds version and immutability data structures to the exception lists and the value lists.
Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number.
Does not add a version and immutability to the exception list items and value list items.
Does not update the version number when you add a new exception list item or value list item.

Examples:

❯ ./post_list.sh

{
  "_version": "WzAsMV0=",
  "id": "ip_list",
  "created_at": "2020-07-21T20:31:11.679Z",
  "created_by": "yo",
  "description": "This list describes bad internet ip",
  "immutable": false,
  "name": "Simple list with an ip",
  "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
  "type": "ip",
  "updated_at": "2020-07-21T20:31:11.679Z",
  "updated_by": "yo",
  "version": 1
}

❯ ./post_exception_list.sh

{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "WzMzOTgsMV0=",
  "created_at": "2020-07-21T20:31:35.933Z",
  "created_by": "yo",
  "description": "This is a sample endpoint type exception",
  "id": "2c24b100-cb91-11ea-a872-adfddf68361e",
  "immutable": false,
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
  "type": "detection",
  "updated_at": "2020-07-21T20:31:35.952Z",
  "updated_by": "yo",
  "version": 1
}

❯ ./update_list.sh
{
  "_version": "WzEsMV0=",
  "created_at": "2020-07-21T20:31:11.679Z",
  "created_by": "yo",
  "description": "Some other description here for you",
  "id": "ip_list",
  "immutable": false,
  "name": "Changed the name here to something else",
  "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5",
  "type": "ip",
  "updated_at": "2020-07-21T20:31:47.089Z",
  "updated_by": "yo",
  "version": 2
}

❯ ./update_exception_list.sh
{
  "_tags": [
    "endpoint",
    "process",
    "malware",
    "os:linux"
  ],
  "_version": "WzMzOTksMV0=",
  "created_at": "2020-07-21T20:31:35.933Z",
  "created_by": "yo",
  "description": "Different description",
  "id": "2c24b100-cb91-11ea-a872-adfddf68361e",
  "immutable": false,
  "list_id": "simple_list",
  "name": "Sample Endpoint Exception List",
  "namespace_type": "single",
  "tags": [
    "user added string for a tag",
    "malware"
  ],
  "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f",
  "type": "endpoint",
  "updated_at": "2020-07-21T20:31:56.628Z",
  "updated_by": "yo",
  "version": 2
}

Checklist

Unit or functional tests were updated or added to match the most common scenarios

…lize immutable lists

elasticmachine · 2020-07-21T19:54:28Z

Pinging @elastic/siem (Team:SIEM)

kibanamachine · 2020-07-21T23:00:03Z

💛 Build succeeded, but was flaky

continuous-integration/kibana-ci/pull-request
Commit: ea9ebdf
Flaky suites:
- xpack-securitySolutionCypress

Test Failures

Kibana Pipeline / kibana-xpack-agent / sets the flyout button background to euiColorSuccess with a 20% alpha channel when the user starts dragging a host, but is not hovering over the flyout button.timeline flyout button sets the flyout button background to euiColorSuccess with a 20% alpha channel when the user starts dragging a host, but is not hovering over the flyout button

Link to Jenkins

Stack Trace

AssertionError: Timed out retrying: expected '<button.euiButton.euiButton--primary.euiButton--iconRight.timeline-flyout-button>' to have CSS property 'background' with the value 'rgba(1, 125, 115, 0.2) none repeat scroll 0% 0% / auto padding-box border-box', but the value was 'rgb(250, 251, 253) none repeat scroll 0% 0% / auto padding-box border-box'
    at applyChainer (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:131898:32)
    at http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:131940:16
    at arrayReduce (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:18578:21)
    at Function.reduce (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:27576:14)
    at applyChainers (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:131929:22)
    at tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
    at Function.Promise.attempt.Promise.try (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:6339:29)
    at Context.shouldFn (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:131946:26)
    at Context.should (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:131964:23)
    at http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:127819:39
    at assertions (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:128079:14)
    at tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
    at Object.gotValue (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:8209:18)
    at Object.gotAccum (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:8196:25)
    at Object.tryCatcher (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:9065:23)
    at Promise._settlePromiseFromHandler (http://elastic:changeme@localhost:61141/__cypress/runner/cypress_runner.js:7000:31)

Build metrics

page load bundle size

id	value	diff	baseline
lists	257.8KB	+1.1KB	256.7KB
securitySolution	870.6KB	+355.0B	870.2KB
total	-	+1.5KB	-

Saved Objects .kibana field count

id	value	diff	baseline
exception-list	37	+2	35
exception-list-agnostic	37	+2	35
total	-	+4	-

History

💔 Build #63414 failed aa3b4cc

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

yctercero · 2020-07-21T23:33:21Z

x-pack/plugins/lists/common/schemas/request/create_list_schema.ts

-  t.exact(t.partial({ deserializer, id, meta, serializer })),
+  t.exact(
+    t.partial({
+      deserializer, // defaults to undefined if not set during decode


These comments have been super helpful throughout the lists api fyi - thank you!

yctercero · 2020-07-21T23:34:45Z

x-pack/plugins/lists/common/schemas/request/patch_list_schema.ts

-  t.exact(t.partial({ _version, description, meta, name })),
+  t.exact(
+    t.partial({
+      _version, // is undefined if not set during decode


Is the _version the immutable/internal version for our use? Then user can set it's own version if it likes for version?

The _version is for the optimistic concurrency control and to get conflicts from an earlier PR of here:
#72337

The version is for when we have immutable lists and want to version those to know which ones are loaded like rules are.

So the UI submits _version if it wants optimistic concurrency control, and then version will be used for immutable lists and the UI probably won't ever have to worry about sending it down or doing anything with it.

yctercero

LGTM! Super clean and easy to go through. My only comment would be if we would want to associate the item with the list version? So you could say, pull all the items from list version x or something. Just a curiosity.

yctercero · 2020-07-21T23:39:13Z

x-pack/plugins/lists/server/services/exception_lists/utils.ts

@@ -108,6 +111,7 @@ export const transformSavedObjectToExceptionList = ({
    type: exceptionListType.is(type) ? type : 'detection',
    updated_at: updatedAt ?? dateNow,
    updated_by,
+    version: version ?? 1, // This should never be undefined for a list (only a list item)


FrankHassanabad · 2020-07-21T23:48:11Z

My only comment would be if we would want to associate the item with the list version? So you could say, pull all the items from list version x or something. Just a curiosity.

At the moment there is only 1 version of any list so if you change the name, description, etc.. of a list then that list bumps its version number and only that one list exists. There is not multiple versions of lists.

…ructures (elastic#72730) ### Summary The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures. * Adds version and immutability data structures to the exception lists and the value lists. * Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number. * Does _not_ add a version and immutability to the exception list items and value list items. * Does _not_ update the version number when you add a new exception list item or value list item. **Examples:** ❯ ./post_list.sh ```json { "_version": "WzAsMV0=", "id": "ip_list", "created_at": "2020-07-21T20:31:11.679Z", "created_by": "yo", "description": "This list describes bad internet ip", "immutable": false, "name": "Simple list with an ip", "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5", "type": "ip", "updated_at": "2020-07-21T20:31:11.679Z", "updated_by": "yo", "version": 1 } ``` ❯ ./post_exception_list.sh ```json { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzMzOTgsMV0=", "created_at": "2020-07-21T20:31:35.933Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "2c24b100-cb91-11ea-a872-adfddf68361e", "immutable": false, "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f", "type": "detection", "updated_at": "2020-07-21T20:31:35.952Z", "updated_by": "yo", "version": 1 } ``` ```json ❯ ./update_list.sh { "_version": "WzEsMV0=", "created_at": "2020-07-21T20:31:11.679Z", "created_by": "yo", "description": "Some other description here for you", "id": "ip_list", "immutable": false, "name": "Changed the name here to something else", "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5", "type": "ip", "updated_at": "2020-07-21T20:31:47.089Z", "updated_by": "yo", "version": 2 } ``` ```json ❯ ./update_exception_list.sh { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzMzOTksMV0=", "created_at": "2020-07-21T20:31:35.933Z", "created_by": "yo", "description": "Different description", "id": "2c24b100-cb91-11ea-a872-adfddf68361e", "immutable": false, "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f", "type": "endpoint", "updated_at": "2020-07-21T20:31:56.628Z", "updated_by": "yo", "version": 2 } ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

…ructures (#72730) (#72774) ### Summary The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures. * Adds version and immutability data structures to the exception lists and the value lists. * Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number. * Does _not_ add a version and immutability to the exception list items and value list items. * Does _not_ update the version number when you add a new exception list item or value list item. **Examples:** ❯ ./post_list.sh ```json { "_version": "WzAsMV0=", "id": "ip_list", "created_at": "2020-07-21T20:31:11.679Z", "created_by": "yo", "description": "This list describes bad internet ip", "immutable": false, "name": "Simple list with an ip", "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5", "type": "ip", "updated_at": "2020-07-21T20:31:11.679Z", "updated_by": "yo", "version": 1 } ``` ❯ ./post_exception_list.sh ```json { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzMzOTgsMV0=", "created_at": "2020-07-21T20:31:35.933Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "2c24b100-cb91-11ea-a872-adfddf68361e", "immutable": false, "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f", "type": "detection", "updated_at": "2020-07-21T20:31:35.952Z", "updated_by": "yo", "version": 1 } ``` ```json ❯ ./update_list.sh { "_version": "WzEsMV0=", "created_at": "2020-07-21T20:31:11.679Z", "created_by": "yo", "description": "Some other description here for you", "id": "ip_list", "immutable": false, "name": "Changed the name here to something else", "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5", "type": "ip", "updated_at": "2020-07-21T20:31:47.089Z", "updated_by": "yo", "version": 2 } ``` ```json ❯ ./update_exception_list.sh { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzMzOTksMV0=", "created_at": "2020-07-21T20:31:35.933Z", "created_by": "yo", "description": "Different description", "id": "2c24b100-cb91-11ea-a872-adfddf68361e", "immutable": false, "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f", "type": "endpoint", "updated_at": "2020-07-21T20:31:56.628Z", "updated_by": "yo", "version": 2 } ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

…ructures (#72730) (#72773) ### Summary The intent is to get the data structures in similar to rules so that we can have eventually immutable and versioned lists in later releases without too much hassle of upgrading the list and list item data structures. * Adds version and immutability data structures to the exception lists and the value lists. * Adds an optional version number to the update route of each so that you can modify the number either direction or you can omit it and it works like the detection rules where it will auto-increment the number. * Does _not_ add a version and immutability to the exception list items and value list items. * Does _not_ update the version number when you add a new exception list item or value list item. **Examples:** ❯ ./post_list.sh ```json { "_version": "WzAsMV0=", "id": "ip_list", "created_at": "2020-07-21T20:31:11.679Z", "created_by": "yo", "description": "This list describes bad internet ip", "immutable": false, "name": "Simple list with an ip", "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5", "type": "ip", "updated_at": "2020-07-21T20:31:11.679Z", "updated_by": "yo", "version": 1 } ``` ❯ ./post_exception_list.sh ```json { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzMzOTgsMV0=", "created_at": "2020-07-21T20:31:35.933Z", "created_by": "yo", "description": "This is a sample endpoint type exception", "id": "2c24b100-cb91-11ea-a872-adfddf68361e", "immutable": false, "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f", "type": "detection", "updated_at": "2020-07-21T20:31:35.952Z", "updated_by": "yo", "version": 1 } ``` ```json ❯ ./update_list.sh { "_version": "WzEsMV0=", "created_at": "2020-07-21T20:31:11.679Z", "created_by": "yo", "description": "Some other description here for you", "id": "ip_list", "immutable": false, "name": "Changed the name here to something else", "tie_breaker_id": "d6bd7552-84d1-4f95-88c4-cc504517b4e5", "type": "ip", "updated_at": "2020-07-21T20:31:47.089Z", "updated_by": "yo", "version": 2 } ``` ```json ❯ ./update_exception_list.sh { "_tags": [ "endpoint", "process", "malware", "os:linux" ], "_version": "WzMzOTksMV0=", "created_at": "2020-07-21T20:31:35.933Z", "created_by": "yo", "description": "Different description", "id": "2c24b100-cb91-11ea-a872-adfddf68361e", "immutable": false, "list_id": "simple_list", "name": "Sample Endpoint Exception List", "namespace_type": "single", "tags": [ "user added string for a tag", "malware" ], "tie_breaker_id": "c11c4d53-d0be-4904-870e-d33ec7ca387f", "type": "endpoint", "updated_at": "2020-07-21T20:31:56.628Z", "updated_by": "yo", "version": 2 } ``` ### Checklist - [x] [Unit or functional tests](https://github.com/elastic/kibana/blob/master/CONTRIBUTING.md#cross-browser-compatibility) were updated or added to match the most common scenarios

* master: (23 commits) Stabilize closing toast (elastic#72097) stabilize failing test (elastic#72086) Stabilize filter bar test (elastic#72032) Unskip vislib tests (elastic#71452) [ML] Fix layout of anomaly chart tooltip for long field values (elastic#72689) fix preAuth/preRouting mocks (elastic#72663) [Security Solution] Hide KQL bar (all pages) and alerts filters (Detections) when Resolver is full screen (elastic#72788) [Uptime] Rename Whitelist to Allowlist in parse_filter_map (elastic#71584) [Security Solution] Fixes exception modal not loading content (elastic#72770) [Security Solution][Exceptions] - Require non empty entries and non empty string values in exception list items (elastic#72748) [Detections] Add validation for Threshold value field (elastic#72611) [SIEM][Detection Engine][Lists] Adds version and immutability data structures (elastic#72730) [Security Solution][Detections] Validate file type of value lists (elastic#72746) [pre-req] New Component Layout proposal (elastic#72385) [ML] do not throw an error when agg is not supported by UI (elastic#72685) [Resolver] Origin process (elastic#72382) [Ingest Manager] Allow to force unenroll from the UI (elastic#72386) skip 6.8 branch when triggering baseline-capture builds (elastic#72706) [CI] In-progress PR comments (elastic#72211) Fix sorting of scripted string fields (elastic#72681) ...

* master: (34 commits) Adds Role Based Access-Control to the Alerting & Action plugins based on Kibana Feature Controls (elastic#67157) [Monitoring] Revert direct shipping code (elastic#72505) Use server basepath when creating reporting jobs (elastic#72722) Adding api test for transaction_groups /breakdown and /avg_duration_by_browser (elastic#72623) [Task Manager] Addresses flaky test introduced by buffered store (elastic#72815) [Observability] filter "hasData" api by processor event (elastic#72810) do not pass title as part of tsvb request (elastic#72619) [Lens] Legend config (elastic#70619) Stabilize closing toast (elastic#72097) stabilize failing test (elastic#72086) Stabilize filter bar test (elastic#72032) Unskip vislib tests (elastic#71452) [ML] Fix layout of anomaly chart tooltip for long field values (elastic#72689) fix preAuth/preRouting mocks (elastic#72663) [Security Solution] Hide KQL bar (all pages) and alerts filters (Detections) when Resolver is full screen (elastic#72788) [Uptime] Rename Whitelist to Allowlist in parse_filter_map (elastic#71584) [Security Solution] Fixes exception modal not loading content (elastic#72770) [Security Solution][Exceptions] - Require non empty entries and non empty string values in exception list items (elastic#72748) [Detections] Add validation for Threshold value field (elastic#72611) [SIEM][Detection Engine][Lists] Adds version and immutability data structures (elastic#72730) ...

elasticmachine · 2021-09-23T14:29:10Z

Pinging @elastic/security-solution (Team: SecuritySolution)

Adds version and immutablity data structures so we can eventually uti…

13d71b1

…lize immutable lists

FrankHassanabad requested review from a team as code owners July 21, 2020 19:53

FrankHassanabad self-assigned this Jul 21, 2020

FrankHassanabad added v8.0.0 v7.10.0 v7.9.0 Team:SIEM labels Jul 21, 2020

FrankHassanabad added the release_note:enhancement label Jul 21, 2020

FrankHassanabad added 2 commits July 21, 2020 13:55

Merge branch 'master' into add-version

aa3b4cc

Fixes unit test

ea9ebdf

yctercero reviewed Jul 21, 2020

View reviewed changes

yctercero approved these changes Jul 21, 2020

View reviewed changes

FrankHassanabad merged commit eddc62a into elastic:master Jul 21, 2020

FrankHassanabad deleted the add-version branch July 21, 2020 23:50

This was referenced Jul 21, 2020

[7.x] [SIEM][Detection Engine][Lists] Adds version and immutability data structures (#72730) #72773

Merged

[7.9] [SIEM][Detection Engine][Lists] Adds version and immutability data structures (#72730) #72774

Merged

FrankHassanabad mentioned this pull request Aug 4, 2020

[Docs]Detection exceptions and lists APIs elastic/security-docs#35

Closed

MindyRS added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 23, 2021

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[SIEM][Detection Engine][Lists] Adds version and immutability data structures #72730

[SIEM][Detection Engine][Lists] Adds version and immutability data structures #72730

FrankHassanabad commented Jul 21, 2020 •

edited

Loading

elasticmachine commented Jul 21, 2020

kibanamachine commented Jul 21, 2020

Stack Trace

yctercero Jul 21, 2020

yctercero Jul 21, 2020

FrankHassanabad Jul 21, 2020 •

edited

Loading

yctercero left a comment

yctercero Jul 21, 2020

FrankHassanabad commented Jul 21, 2020

elasticmachine commented Sep 23, 2021

[SIEM][Detection Engine][Lists] Adds version and immutability data structures #72730

[SIEM][Detection Engine][Lists] Adds version and immutability data structures #72730

Conversation

FrankHassanabad commented Jul 21, 2020 • edited Loading

Summary

Checklist

elasticmachine commented Jul 21, 2020

kibanamachine commented Jul 21, 2020

💛 Build succeeded, but was flaky

Test Failures

Stack Trace

Build metrics

page load bundle size

Saved Objects .kibana field count

History

yctercero Jul 21, 2020

Choose a reason for hiding this comment

yctercero Jul 21, 2020

Choose a reason for hiding this comment

FrankHassanabad Jul 21, 2020 • edited Loading

Choose a reason for hiding this comment

yctercero left a comment

Choose a reason for hiding this comment

yctercero Jul 21, 2020

Choose a reason for hiding this comment

FrankHassanabad commented Jul 21, 2020

elasticmachine commented Sep 23, 2021

FrankHassanabad commented Jul 21, 2020 •

edited

Loading

FrankHassanabad Jul 21, 2020 •

edited

Loading