Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: Rules Management - Rule Bulk Actions Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_ess·ts - Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic should disable rules and migrate actions #196462

Open
2 of 3 tasks
Tracked by #209132
kibanamachine opened this issue Oct 16, 2024 · 25 comments · Fixed by #197660
Open
2 of 3 tasks
Tracked by #209132

Failing test: Rules Management - Rule Bulk Actions Integration Tests - ESS Env - Trial License.x-pack/test/security_solution_api_integration/test_suites/detections_response/rules_management/rule_bulk_actions/trial_license_complete_tier/perform_bulk_action_ess·ts - Rules Management - Rule Bulk Action API @ess perform_bulk_action - ESS specific logic should disable rules and migrate actions #196462

kibanamachine opened this issue Oct 16, 2024 · 25 comments · Fixed by #197660
Assignees
@maximpn
Labels
8.18 candidate blocker failed-test A test failure on a tracked branch, potentially flaky-test Feature:Rule Management Security Solution Detection Rule Management area skipped-test Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.19.0 v9.1.0

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Oct 16, 2024 •
edited by banderror
Loading

A test failed on a tracked branch

TypeError: Cannot read properties of undefined (reading 'uuid')
    at Context.<anonymous> (perform_bulk_action_ess.ts:209:37)
    at processTicksAndRejections (node:internal/process/task_queues:95:5)
    at Object.apply (wrap_function.js:74:16)

First failure: kibana-on-merge - main

Todo
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Oct 16, 2024
@botelastic botelastic bot added the needs-team Issues missing a team label label Oct 16, 2024
@kibanamachine kibanamachine added the Team:Detection Rule Management Security Detection Rule Management Team label Oct 16, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detection-rule-management (Team:Detection Rule Management)

@botelastic botelastic bot removed the needs-team Issues missing a team label label Oct 16, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@jbudz
Copy link
Member

jbudz commented Oct 16, 2024

/skip

kibanamachine added a commit that referenced this issue Oct 16, 2024
@kibanamachine
skip failing test suite (#196462)
9bca8b7
@kibanamachine
Copy link
Contributor Author

Skipped

main: 9bca8b7
8.x: 5f86c95

kibanamachine added a commit that referenced this issue Oct 16, 2024
@kibanamachine
skip failing test suite (#196462)
5f86c95
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@banderror banderror added Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. Feature:Rule Management Security Solution Detection Rule Management area 8.16 candidate labels Oct 22, 2024
@elasticmachine
Copy link
Contributor

Pinging @elastic/security-solution (Team: SecuritySolution)

@elasticmachine
Copy link
Contributor

Pinging @elastic/security-detections-response (Team:Detections and Resp)

@banderror
Copy link
Contributor

@maximpn @xcrzx 👋

I assigned you to this one and #196470. By the end of this week we need to figure out who can work on these. The goal is to understand if these test failures are indications of a bug - in which case we will have less than one week for fixing it for v8.16.0 according to the release schedule.

@maximpn maximpn mentioned this issue Oct 28, 2024
Merged
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (elastic…
34cd80b 
…#197660)

**Closes: elastic#196470
**Closes: elastic#196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages.

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢

(cherry picked from commit bf37a01)
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Nov 5, 2024
@maximpn
[Security Solution] Unskip perform bulk action ESS FTR tests (elastic…
3792382 
…#197660)

**Closes: elastic#196470
**Closes: elastic#196462

## Summary

This PR unskips `perform_bulk_action_ess.ts` functional test.

## Details

`perform_bulk_action_ess.ts` includes a number of functional tests where some of them were flaky. Investigation revealed that creating enabled rules and performing bulk actions may lead to race conditions. Under that conditions rule's SO isn't updated as expected. For example legacy rule actions aren't persisted in rule's SO when it's expected by the test.

This PR includes `perform_bulk_action_ess.ts` refactoring to create disabled rules in the majority of tests. Enabled rules are created only in tests checking behavior upon rules disabling. These tests were checked multiple times and didn't demonstrate flakiness.

Additionally `@kbn/expect` was replaced with `expect` to make asserting more transparent and avoid unclear error messages.

## Flaky test results

[100 runs](https://buildkite.com/elastic/kibana-flaky-test-suite-runner/builds/7274) 🟢

(cherry picked from commit bf37a01)
@kibanamachine kibanamachine reopened this Nov 5, 2024
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

@maximpn maximpn mentioned this issue Nov 6, 2024
Merged
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.16

maximpn added a commit that referenced this issue Nov 7, 2024
@maximpn
[8.16] [Security Solution] Skip a flaky legacy actions migration upon…
e29cc5f 
… bulk action FTR test (#199200)

**Addresses:** #196462

## Summary

`perform_bulk_action_ess.ts`'s `ESS specific logic` -> `should disable rules and migrate actions` FTR test is flaky in `8.16`. This skips only this particular test.

## More details on flakiness

Flakiness isn't reproducible in `main` which is `v9.0` currently.  It's suspected that modification of rule's SO interferences with rules bulk action performing a legacy action migration. Though a quick fix like waiting for rule execution completion before performing bulk action didn't help. It requires further investigation.

Another idea `8.16` could require some PR from to be backported which fixed this issue in `main`.
@banderror banderror mentioned this issue Jan 31, 2025
Open
@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

kibanamachine added a commit that referenced this issue Feb 4, 2025
@kibanamachine
skip failing test suite (#196462)
3e147d5
kibanamachine added a commit that referenced this issue Feb 4, 2025
@kibanamachine
skip failing test suite (#196462)
b458385
@kibanamachine
Copy link
Contributor Author

Skipped

main: 3e147d5
8.x: b458385

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - 8.x

@kibanamachine
Copy link
Contributor Author

New failure: kibana-on-merge - main

@nikitaindik
Copy link
Contributor

We've taken a look at it together with @maximpn. The failure is in a test that checks that disabling a rule with legacy actions migrates these actions.

This needs to be investigated more thoroughly to find out the reason.

Image

As a stopgap solution @maximpn will split this test suite into multiple, and push the changes to pinpoint the tests to be skipped. He'll take care of it later once he has more cycles.

drewdaemon pushed a commit to drewdaemon/kibana that referenced this issue Feb 6, 2025
@kibanamachine @drewdaemon
skip failing test suite (elastic#196462)
018e33b
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@maximpn maximpn

Labels
8.18 candidate blocker failed-test A test failure on a tracked branch, potentially flaky-test Feature:Rule Management Security Solution Detection Rule Management area skipped-test Team:Detection Rule Management Security Detection Rule Management Team Team:Detections and Resp Security Detection Response Team Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. v8.19.0 v9.1.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging a pull request may close this issue.

[Security Solution] Unskip perform bulk action ESS FTR tests maximpn/kibana
7 participants
@xcrzx @jbudz @maximpn @banderror @elasticmachine @nikitaindik @kibanamachine