Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/ip_array·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type ip "is in list" operator will return 1 result if we have a list that includes all ips #89052

Closed
kibanamachine opened this issue Jan 22, 2021 · 3 comments
Closed

Failing test: X-Pack Detection Engine API Integration Tests.x-pack/test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/ip_array·ts - detection engine api security and spaces enabled Detection exceptions data types and operators Rule exception operators for data type ip "is in list" operator will return 1 result if we have a list that includes all ips #89052

kibanamachine opened this issue Jan 22, 2021 · 3 comments
Assignees
@FrankHassanabad
Labels
failed-test A test failure on a tracked branch, potentially flaky-test skipped-test Team:Detections and Resp Security Detection Response Team

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Jan 22, 2021 •
edited
Loading

A test failed on a tracked branch

Error: expected [] to sort of equal [ [] ]
    at Assertion.assert (/dev/shm/workspace/parallel/8/kibana/packages/kbn-expect/expect.js:100:11)
    at Assertion.eql (/dev/shm/workspace/parallel/8/kibana/packages/kbn-expect/expect.js:244:8)
    at Context.<anonymous> (test/detection_engine_api_integration/security_and_spaces/tests/exception_operators_data_types/ip_array.ts:506:24)
    at Object.apply (/dev/shm/workspace/parallel/8/kibana/packages/kbn-test/src/functional_test_runner/lib/mocha/wrap_function.js:73:16) {
  actual: '[]',
  expected: '[\n  []\n]',
  showDiff: true
}

First failure: Jenkins Build
@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Jan 22, 2021
@mikecote mikecote added Team:Detections and Resp Security Detection Response Team triage_needed labels Jan 22, 2021
@kibanamachine
Copy link
Contributor Author

New failure: CI Build - master

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - master

jbudz added a commit that referenced this issue Oct 17, 2021
@jbudz
skip flaky tests. #89052, #113418, #115304
06e66ca
jbudz added a commit that referenced this issue Oct 17, 2021
@jbudz
skip flaky tests. #89052, #113418, #115304
f002f58
@jbudz
Copy link
Member

jbudz commented Oct 17, 2021

skipped

main: 06e66ca
7.x: f002f58

jloleysens added a commit to jloleysens/kibana that referenced this issue Oct 18, 2021
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into reporting/poc…
6b639dd 
…-migrate-away-from-injected-css-js

* 'master' of github.com:elastic/kibana: (237 commits)
  [Uptime] Added uptime query inspector panel (elastic#115170)
  [Osquery] Add packs (elastic#107345)
  [App Search] Allow for query parameter to indicate ingestion mechanism for new engines (elastic#115188)
  [Alerting] Active alerts do not recover after re-enabling a rule (elastic#111671)
  skip flaky tests.  elastic#115308, elastic#115313
  [Breaking] Remove deprecated `enabled` settings from plugins. (elastic#113495)
  skip flaky suite.  elastic#107057
  skip flaky tests. elastic#89052, elastic#113418, elastic#115304
  skip flaky test. elastic#113892
  Bump node to 16.11.1 (elastic#110684)
  [Security Solution] Restores Alerts table local storage persistence and the Remove Column action (elastic#114742)
  skip flaky suite.  elastic#115130
  one line remove assert (elastic#115127)
  Fixes migration bug where I was deleting attributes (elastic#115098)
  [Security Solutions] Fixes the newer notification system throttle resets and enabling immediate execution on first detection of a signal  (elastic#114214)
  [build] Dockerfile update (elastic#115237)
  Fixes Cypress flake cypress test (elastic#115270)
  Disable APM e2e tests
  log an invalid type for SO (elastic#115175)
  [Fleet] Don't auto upgrade policies for AUTO_UPDATE packages (elastic#115199)
  ...

# Conflicts:
#	src/plugins/dashboard/public/application/dashboard_app.tsx
#	src/plugins/dashboard/public/types.ts
#	x-pack/plugins/reporting/server/lib/layouts/print_layout.ts
jloleysens added a commit to jloleysens/kibana that referenced this issue Oct 18, 2021
@jloleysens
Merge branch 'master' of github.com:elastic/kibana into reporting/add…
36426c6 
…-link-to-kibana-app

* 'master' of github.com:elastic/kibana: (287 commits)
  [Security Solution][Endpoint] Change `trustedAppByPolicyEnabled` flag to `true` by default (elastic#115264)
  [APM] generator: support error events and application metrics (elastic#115311)
  [kibanaUtils] Don't import full `semver` client side (elastic#114986)
  [RAC] Link inventory alerts to the right inventory view (elastic#113553)
  [Uptime] Added uptime query inspector panel (elastic#115170)
  [Osquery] Add packs (elastic#107345)
  [App Search] Allow for query parameter to indicate ingestion mechanism for new engines (elastic#115188)
  [Alerting] Active alerts do not recover after re-enabling a rule (elastic#111671)
  skip flaky tests.  elastic#115308, elastic#115313
  [Breaking] Remove deprecated `enabled` settings from plugins. (elastic#113495)
  skip flaky suite.  elastic#107057
  skip flaky tests. elastic#89052, elastic#113418, elastic#115304
  skip flaky test. elastic#113892
  Bump node to 16.11.1 (elastic#110684)
  [Security Solution] Restores Alerts table local storage persistence and the Remove Column action (elastic#114742)
  skip flaky suite.  elastic#115130
  one line remove assert (elastic#115127)
  Fixes migration bug where I was deleting attributes (elastic#115098)
  [Security Solutions] Fixes the newer notification system throttle resets and enabling immediate execution on first detection of a signal  (elastic#114214)
  [build] Dockerfile update (elastic#115237)
  ...

# Conflicts:
#	x-pack/plugins/reporting/public/management/__snapshots__/report_listing.test.tsx.snap
@FrankHassanabad FrankHassanabad mentioned this issue Oct 20, 2021
Merged
1 task
@FrankHassanabad FrankHassanabad self-assigned this Oct 20, 2021
kibanamachine pushed a commit to kibanamachine/kibana that referenced this issue Oct 20, 2021
@FrankHassanabad @kibanamachine
[Security Solutions] Fixes 11 different flakey FTR/e2e tests and scen…
711c67c 
…arios (elastic#115688)

## Summary

Fixes flakes across tests that have either been skipped or have been a source of flake in the categories of:
* Sorting fixes because Elasticsearch can return hits/arrays back in different orders
* Flat array fixes because Elasticsearch can sometimes return `[]` or `[[]]` in-deterministically in some cases 🤷 , so we just flatten the array out completely and test for `[]` within those tests.
* `waitForSignalsToBePresent` was missing in a test and sometimes we would get an empty array response which would fail CI.

Also I audited other tests for `[[]]` and `waitForSignalsToBePresent` and fixed them where they were present or if the `waitForSignalsToBePresent` count was incorrect. This should give us more stability when the CI is under pressure.

Sorting fixes:
elastic#115554
elastic#115321
elastic#115319
elastic#114581


Flat array fixes:
elastic#89052
elastic#115315
elastic#115308
elastic#115304
elastic#115313
elastic#113418

Missing additional check for "waitForSignalsToBePresent" or incorrect number of signals to wait for fixes:
elastic#115310


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
kibanamachine added a commit that referenced this issue Oct 20, 2021
@kibanamachine @FrankHassanabad
[Security Solutions] Fixes 11 different flakey FTR/e2e tests and scen…
cd613d3 
…arios (#115688) (#115699)

## Summary

Fixes flakes across tests that have either been skipped or have been a source of flake in the categories of:
* Sorting fixes because Elasticsearch can return hits/arrays back in different orders
* Flat array fixes because Elasticsearch can sometimes return `[]` or `[[]]` in-deterministically in some cases 🤷 , so we just flatten the array out completely and test for `[]` within those tests.
* `waitForSignalsToBePresent` was missing in a test and sometimes we would get an empty array response which would fail CI.

Also I audited other tests for `[[]]` and `waitForSignalsToBePresent` and fixed them where they were present or if the `waitForSignalsToBePresent` count was incorrect. This should give us more stability when the CI is under pressure.

Sorting fixes:
#115554
#115321
#115319
#114581


Flat array fixes:
#89052
#115315
#115308
#115304
#115313
#113418

Missing additional check for "waitForSignalsToBePresent" or incorrect number of signals to wait for fixes:
#115310


### Checklist

- [x] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios

Co-authored-by: Frank Hassanabad <frank.hassanabad@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@FrankHassanabad FrankHassanabad

Labels
failed-test A test failure on a tracked branch, potentially flaky-test skipped-test Team:Detections and Resp Security Detection Response Team
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests
5 participants
@FrankHassanabad @jbudz @mikecote @yctercero @kibanamachine