[Security Solution] Exploratory testing of prebuilt rule customization workflows

[jpdjere]

Epics: https://github.com/elastic/security-team/issues/1974 (internal), #174168

Summary

Do comprehensive exploratory testing of the app with the prebuiltRulesCustomizationEnabled feature flag turned on.

Workflows to test that were directly affected by the prebuilt rule customization functionality:

• Prebuilt rule customization: installing, editing, bulk editing, duplicating prebuilt rules; using the Rule Management and Details pages with customized and non-customized prebuilt rules, as well as custom rules.
• Prebuilt rule upgrade: upgrading customized and non-customized prebuilt rules.
• Rule export and import: exporting and importing customized and non-customized prebuilt rules, as well as custom rules.

Workflows to test for regressions:

• Rule management: creating, editing, bulk editing, duplicating custom rules; using the Rule Management and Details pages with custom rules.
• Rule installation workflow: installing prebuilt rules; using the Rule Management and Details pages.

Advanced testing:

• Disable the prebuiltRulesCustomizationEnabled feature flag after using the app with the flag enabled. Test that the app works without issues and errors, even there are some customized prebuilt rules that were created when the flag was turned on.

[elasticmachine]

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

[elasticmachine]

Pinging @elastic/security-solution (Team: SecuritySolution)

[elasticmachine]

Pinging @elastic/security-detections-response (Team:Detections and Resp)

[banderror]

@MadameSheema please reassign to someone from QA Source when you know who could help us with this.

[pborgonovi]

Prebuilt Rule Customization Test Plan

1. Introduction
   This feature allows users to customize prebuilt rules, import/export them, and upgrade while preserving customizations. Testing will focus on rule customization, the Three-Way-Diff component, conflict resolution, bulk upgrades, and import/export functionality to ensure a reliable and seamless user experience.

2. Scope of Testing
   2.1 In-Scope
   Full Integration System Testing.

Feature	Priority	Risk Level	Testing Conditions	Comments
Prebuilt Rules Customization	High	High	Test that prebuilt rules can be customized and upgraded correctly while preserving user modifications.	Validate that updates can be done in bulk and that rule integrity is maintained.
Rule Management Table Updates	Moderate	Moderate	Test UI functionality for filtering, selecting, and sorting rules; verify distinction between Prebuilt, Customized, and Custom rules.	Ensure the UI behaves correctly with bulk actions and different rule types.
Bulk Rule Upgrades	High	High	Test that bulk upgrades process only conflict-free rules, while conflicts are managed and excluded with clear notifications.	Validate this workflow thoroughly, as it’s highly impacted by the new changes.
Upgrade Flyout and Conflict Review	High	High	Test the Three-Way-Diff component’s handling of conflicts during upgrades; verify user interactions like accepting, editing, and saving changes.	Ensure accurate display of field-by-field differences and verify that conflicts are resolved properly for both solvable and non-solvable cases.
Auto-Merge of Customizations	High	High	Test auto-merge functionality to ensure that changes from different versions are combined correctly during an upgrade.	Focus on preserving customizations while applying new updates, ensuring merged results reflect user preferences without altering rule integrity.
Rule Type Changes	High	High	Test rule type changes during upgrades, confirming user warnings and allowing users to clone or retain customizations.	Ensure UI handles type changes clearly, warning users about potential data loss or incompatible changes.
Import/Export of Customized Rules	Moderate	Moderate	Test import/export functionality to ensure that customizations are preserved when moving rules between instances. Test import/export functionality between ESS and Serverless instances.	Validate behavior when handling multiple versions of the same rule, ensuring customizations are retained during re-import.
Rule Execution	Moderate	Moderate	Ensure customized and upgraded rules execute correctly, triggering the intended detections and maintaining functionality.	This can be tested as part of regression testing, as changes should minimally affect execution.
Regression Testing	Moderate	High	Conduct exploratory regression testing to verify that new features do not disrupt existing functionalities.	Test rule installation, duplication, editing of custom rules, and bulk actions; check that turning off the feature flag preserves current functionalities.

2.2 Out-of-Scope

• API testing (since testing will be done exclusively through the UI).
• Performance and load testing

3. Objectives
   • Our main objective is to release this functionality free of significant issues.

4. Test Strategy
   4.1 Testing Approach

   • Exploratory Testing (Risk-Based Priority):
     The primary focus will be on exploratory testing, with testing efforts prioritized according to the risk levels identified.
     High-Risk Areas will receive the most attention to ensure critical functionalities are working correctly.

   Prioritized Areas for Exploratory Testing:

   • High-Risk Areas (Focus First):
     • Prebuilt Rule Customization
     • Bulk Rule Upgrades
     • Rule Type Changes
     • Conflict Resolution Logic (Three-Way-Diff)
     • Auto-Merging of Customizations

   • Moderate-Risk Areas (Secondary Focus):
     • Rule Execution and Alerts
     • Import/Export of Customized Rules
     • Rule Management Table
     • Regression Testing

   • Regression Testing:

     • Conduct exploratory regression testing across key areas to ensure that new features do not break existing functionalities like rule installation, duplication, editing custom rules from both Rule Management and Rule Details page.
     • Turn flag OFF and validate existing functionalities continue to work as expected.

[banderror]

@pborgonovi Great work, and thank you for this initiative. The plan for exploratory testing looks good so far 👍

A few suggestions:

• I would suggest to try to improve the plan's readability. Currently, we repeat pretty much the same thoughts / focus areas multiple times in the "Testing includes", "Objectives", "Prioritized Areas for Exploratory Testing", and "Risk Matrix" sections. Let's try to consolidate this in a single section "Scope", where we'd have a "In-scope" section with a table listing all testing efforts to do, their priority, risk, comments, etc; and a "Out-of-scope" section that could contain a simple list.
• Our objective is to release the feature in time without any significant issues ☝ 🙂
• I would disagree on some of the priorities:
  • "Rule Execution" and "Alerts Based on Customized Rules", while being core features, is a lower risk because we know how it works under the hood, and the new changes shouldn't affect this. Still, worth testing, of course, but could be a part of regression testing with a bit lower priority.
  • "Bulk Rule Upgrades" should be a high risk and high priority. This workflow is highly affected by the new changes.

[pborgonovi]

Thank you so much for the feedback @banderror I'll be working on it.