[Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3

[banderror]

Epic: https://github.com/elastic/security-team/issues/1974 (internal)
Milestones: << • >>

Status: In development.

Summary

Milestone 3: Add support for customizing prebuilt rules. Extend the rule upgrade workflow with full support for 3-way diffs and conflict resolution.

This meta ticket is created to simplify tracking of various tickets related to the epic, and to make this public information so our users can track the progress.

User-facing outcomes:

• Users can click “edit” button for prebuilt rules and customise any field in the same editing interface as the custom rules
• User can filter rules in the rule management page by custom/customised/Elastic
• User can see if the rule was customised on the rule details page
• If the prebuilt rule is customised and update comes in:
  • User can see the current version and update and compare per field
  • They are able to edit the final field versions before finalising the update
  • If the rule type changes - they can only accept the incoming changes
• Prebuilt rules can be exported and imported

Useful info:

• RFC: Prebuilt Rules Customization - a new technical design document describing in detail how prebuilt rule customization should work and affect other existing features.
• Prebuilt Rules Customization Technical Design - an older document from Milestone 1.

Design

Technical design

Beta Give feedback

1. [Security Solution] Write an RFC for customizing prebuilt rules #171309
   8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
2. [Security Solution] Discuss rule fields in the context of the prebuilt rule upgrade workflow #147239
   8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp discuss +6
   

   
3. [Security Solution] Discuss rule field diffs and conflicts to handle in the UI #148913
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp discuss release_note:skip skip-ci +7
   

UI/UX design

Beta Give feedback

1. [Security Solution] Discuss UI/UX designs for customizing prebuilt rules #178211
   8.14 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp design +6
2. Design rule management changes (Rule Management and Details pages)
3. Design rule customization changes (Rule Management and Editing pages)
4. Design rule installation and upgrade changes (Add Rules page, Rule updates table, rule upgrade details flyout)
5. Design rule upgrade UI for rule type changes (implementation ticket)

Preparatory changes

Preparatory changes is something we can work on before starting to hide functionality behind a feature flag. This will reduce the overall complexity introduced by feature toggling.

Missing UI for editing certain rule fields

Beta Give feedback

1. [Security Solution] Allow users to edit setup field for custom rules #173626
   8.14 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.14.0 +8
   
2. [Security Solution] Allow users to edit max_signals field for custom rules #173593
   8.15 candidate Feature:Rule Creation Feature:Rule Details Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +9
   
3. [Security Solution] Allow users to edit related_integrations field for custom rules #173595
   8.15 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +8
   
4. [Security Solution] Allow users to edit required_fields field for custom rules #173594
   8.15 candidate Feature:Rule Creation Feature:Rule Edit Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +8
   

Missing UI for editing certain rule fields (docs)

Beta Give feedback

1. Edit setup guide field for custom rules in UI and API security-docs#4917
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.14.0 +5
   
2. Edit max_signals field for custom rules in UI and API security-docs#5029
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0 +5
   
3. Edit related_integrations field for custom rules in UI and API security-docs#5099
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0 +5
   
4. Edit required_fields field for custom rules in UI and API security-docs#5131
   3 of 3
   Docset: ESS Docset: Serverless Feature: Rules Team: Detections/Response v8.15.0 +5
   

Schema-related changes

Beta Give feedback

1. [Security Solution] Add rule_source to the API schema #180122
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
2. [Security Solution] Make changes in the internal rule schema #180124
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
3. [Security Solution] Replaced the incorrect runtime type used for ruleSource #184004
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp backport:skip bug impact:critical release_note:skip v8.15.0 +9
   
4. [Security Solution] Write the rule_source field together with immutable #180141
   8.15 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +7
   
5. [Security Solution] RulesManagementClient refactoring. Part 1 #180128
   8.15 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring v8.15.0 +7
   
6. [Security Solution] DetectionRulesClient refactoring. Part 2 #184364
   8.16 candidate Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring v8.15.0 v8.16.0 +8
   
7. [Security Solution] Implement normalization on read for rule_source and immutable fields #180140
   8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +7
   

Rule customization, API changes

Beta Give feedback

1. [Security Solution] Calculate and save ruleSource.isCustomized in API endpoint handlers #180145
   8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +7
   
2. [Security Solution] Calculate and save ruleSource.isCustomized in bulk edit API #187706
   8.16 candidate Feature:Prebuilt Detection Rules Feature:Rule Management Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +7
   
3. [Security Solution] Make rule_source field required in RuleResponse #180270
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
4. [Security Solution] Prevent from updating "non-customizable" fields in prebuilt rules #180273
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   

Rule upgrade, API changes

Beta Give feedback

1. [Security Solution] Return user-customized fields from the POST /prebuilt_rules/upgrade/_review API endpoint even if they haven't been updated by Elastic in the target version #180154
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
2. [Security Solution] Extend the POST /upgrade/_review API endpoint's contract and functionality #180153
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
3. [Security Solution] Handle specific fields in /upgrade/_review upgrade workflow #180393
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
4. [Security Solution] Substitute MissingVersion symbol in the ThreeWayDiff object with a boolean #188277
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp refactoring +6
   
5. [Security Solution] Extend the POST /upgrade/_perform API endpoint's contract and functionality #166376
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
6. [Security Solution] Handle specific fields in /upgrade/_perform endpoint upgrade workflow #186544
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
7. [Security Solution] Add InvestigationFields and AlertSuppression fields to the upgrade workflow #190597
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
8. [Security Solution] Remove exceptions_list, author and license from Diffable Rule #196213
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   

Rule upgrade, diff algorithms

Beta Give feedback

1. [Security Solution] Implement number diff algorithm #180160
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +7
   
2. [Security Solution] Implement single-line string diff algorithm #180158
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +7
   
3. [Security Solution] Implement multi-line string diff algorithm #180159
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
4. [Security Solution] Implement array of scalar values diff algorithm #180162
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
5. [Security Solution] Implement data source fields diff algorithm #187659
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
6. [Security Solution] Implement query fields diff algorithms #187658
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   
7. [Security Solution] Implement concurrent_searches and items_per_search fields diff algorithms #188061
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
8. [Security Solution] Implement rule type diff algorithm #190482
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 +7
   

Fleet package with prebuilt rules

Beta Give feedback

1. [Security Solution] OOMs during prebuilt rules package installation #187969
   3 of 3
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet v8.16.0 +7
   
2. [Fleet] All package SOs are being read into memory during package upgrade or force installation #187975
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt +7
   
3. [Fleet] Split bulk delete into chunks to avoid high memory consumption #188208
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet technical debt +7
   
4. [Fleet] Split asset installation into chunks to avoid high memory pressure #189043
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet +6
   
5. [Security Solution] API endpoint for the package with prebuilt rules #187647
   8.16 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 +6
   
6. [Fleet] Stream-based programmatic API for installing packages #187646
   Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet +5
   
7. [Security Solution] Stream-based installation of the package with prebuilt rules #192350
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp Team:Fleet performance v8.17.0 +8
   
8. [Security Solution] Smart limits for the package with prebuilt rules #187645
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.17.0 v8.18.0 +7
   

Changes hidden behind the feature flag

These are changes that will need to be hidden behind the prebuiltRulesCustomizationEnabled feature flag.

Rule customization, UI changes

Beta Give feedback

1. [Security Solution] Add a new feature flag prebuiltRulesCustomizationEnabled #180130
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.15.0 +6
   
2. [Security Solution] Show if a rule is customized on the Rule Details page #180170
   8.15 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.15.0 +7
   
3. [Security Solution] Support filtering by "Modified rules" and "Unmodified rules" #180169
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
4. [Security Solution] Add support for editing prebuilt rules to the Rule Management and Rule Details pages #180171
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
5. [Security Solution] Add support for editing prebuilt rules to the Rule Editing page #180172
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   

Rule upgrade, UI changes

Beta Give feedback

1. [Security Solution] Handle conflicting rules in the Rule Upgrade table #180589
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs design +6
   

   
2. [Security Solution] Rework Update flyout to display all field updates and build Three-Way-Diff field component #171520
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement v8.16.0 v8.17.0 v8.18.0 +9
   
3. [Security Solution] Handle specific fields in the upgrade workflow's UI #188065
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
4. [Security Solution] Implement UI for updating prebuilt rule to a new rule type (MVP) #180395
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
5. [Security Solution] Show Modified badge for fields customized by users #203718
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   

Rule export and import, API and UI changes

Beta Give feedback

1. [Security Solution] Allow exporting prebuilt rules at the API level #180167
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
2. [Security Solution] Allow importing prebuilt rules at the API level #180168
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp enhancement +6
   
3. [Security Solution] Support exporting prebuilt rules from the Rule Management page #180173
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
4. [Security Solution] Support exporting prebuilt rules from the Rule Details page #180176
   8.17 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5
   
5. [Security Solution] Benchmark performance of importing a large number of prebuilt rules #195632
   8.18 candidate Feature:Prebuilt Detection Rules Feature:Rule Import/Export Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +7
   

Licensing

Beta Give feedback

1. https://github.com/elastic/security-team/issues/10410

Telemetry

Beta Give feedback

1. [Security Solution] Collect usage statistics for prebuilt rule customization #140369
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp needs product telemetry +7
   

Before release

Bugs

Beta Give feedback

1. [Security Solution] Users can Customize Prebuilt Detection Rules: Milestone 3 bugs #201502
   29 of 54
   8.18 candidate Feature:Prebuilt Detection Rules Meta Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.16.0 v8.17.0 v8.18.0 +9
   

   

Testing

Beta Give feedback

1. [Security Solution] Write tests for "Customized Elastic rule" badge on the Rule Details page #186916
   8.18 candidate Feature:Prebuilt Detection Rules Feature:Rule Details Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp technical debt test-coverage v8.18.0 +9
   
2. [Security Solution] Tests for prebuilt rule customization workflow #202068
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0 +9
   
3. [Security Solution] Tests for prebuilt rule upgrade workflow #202078
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0 +9
   
4. [Security Solution] Tests for prebuilt rule import/export workflow #202079
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp test test-coverage test-plan v8.18.0 +9
   
5. [Security Solution] Acceptance testing of prebuilt rule customization workflows #180397
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
6. [Security Solution] Exploratory testing of prebuilt rule customization workflows #180398
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   

Documentation

Beta Give feedback

1. [Request] Prebuilt rule customization and upgrade workflows (DRAFT) security-docs#5061
   v8.18.0 +1
   
2. [Request] Document new rule_source property for rules in the API schema (DRAFT) security-docs#5063
   v8.18.0 +1
   

Release

Beta Give feedback

1. [Security Solution] Release prebuilt rule customization feature #180267
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6

After release

Last changes after releasing the feature

Beta Give feedback

1. [Security Solution] Exploratory testing of prebuilt rule customization workflows in production #180399
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp v8.18.0 +6
   
2. [Security Solution] Remove the prebuiltRulesCustomizationEnabled feature flag #180272
   8.18 candidate Feature:Prebuilt Detection Rules Team: SecuritySolution Team:Detection Rule Management Team:Detections and Resp +5

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[banderror]

Draft plan for Milestone 3

UPD: the plan has been moved to the ticket description.