Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Failing test: Jest Tests.x-pack/plugins/security_solution/public/common/hooks - When using the useUpgradeSecurityPackages() hook should call fleet setup first via isInitialized() and then send upgrade request #112910

Closed
kibanamachine opened this issue Sep 23, 2021 · 7 comments · Fixed by #148426
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.

Comments

@kibanamachine
Copy link
Contributor

kibanamachine commented Sep 23, 2021

A test failed on a tracked branch

Error: Timed out in waitFor after 1000ms.
    at Object.waitFor (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/@testing-library/react-hooks/lib/core/asyncUtils.js:70:13)
    at Object.<anonymous> (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/x-pack/plugins/security_solution/public/common/hooks/use_upgrade_secuirty_packages.test.tsx:54:5)
    at _callCircusTest (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:212:5)
    at _runTest (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:149:3)
    at _runTestsForDescribeBlock (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:63:9)
    at _runTestsForDescribeBlock (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:57:9)
    at run (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/run.js:25:3)
    at runAndTransformResultsToJestFormat (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapterInit.js:176:21)
    at jestAdapter (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-circus/build/legacy-code-todo-rewrite/jestAdapter.js:109:19)
    at runTestInternal (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-runner/build/runTest.js:380:16)
    at runTest (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-runner/build/runTest.js:472:34)
    at Object.worker (/opt/local-ssd/buildkite/builds/kb-c2-16-333887c59d35c448/elastic/kibana-hourly/kibana/node_modules/jest-runner/build/testWorker.js:133:12)

First failure: CI Build

@kibanamachine kibanamachine added the failed-test A test failure on a tracked branch, potentially flaky-test label Sep 23, 2021
@botelastic botelastic bot added the needs-team Issues missing a team label label Sep 23, 2021
@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@kibanamachine
Copy link
Contributor Author

New failure: CI Build

@stratoula stratoula added the Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc. label Sep 29, 2021
@botelastic botelastic bot removed the needs-team Issues missing a team label label Sep 29, 2021
mistic added a commit that referenced this issue Oct 1, 2021
mistic added a commit that referenced this issue Oct 1, 2021
@mistic
Copy link
Member

mistic commented Oct 1, 2021

The test looks flaky. Skipping for now.

master/8.0: a4903ac
7.x/7.16: 17a4b74

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 7.15

@kibanamachine
Copy link
Contributor Author

New failure: CI Build - 7.15

@jbudz
Copy link
Member

jbudz commented Oct 17, 2021

7.15: 5a9e711

@spong
Copy link
Member

spong commented Jan 10, 2023

To be un-skipped in #148426. Will run against the flakey test runner beforehand to ensure flake has been resolved 👍

edit: Looks like the flakey test runner doesn't work with jest tests, just FTR's? Will follow this and see if the flake has been resolved then

spong added a commit that referenced this issue Jan 17, 2023
…rules (#148426)

## Summary

Resolves #147466
Resolves #112910

* Updates `useUpgradeSecurityPackages` hook to install the `prerelease` version of the `endpoint` and `security_detection_engine` packages if the current branch is `main` or build is `-SNAPSHOT` (to ensure PR's are testing against the latest to-be-released packages)
* Adds new `kibana.yml` configuration `xpack.securitySolution.prebuiltRulesPackageVersion` for specifying the version of the `security_detection_engine` package to install within the client-side logic of the `useUpgradeSecurityPackages` hook
* Adds FTR helpers for consuming the `xpack.securitySolution.prebuiltRulesPackageVersion` configuration from the `kbnServerArgs` and for installing a specific detection rules package version [c467762](c467762).
* Regenerated docs
* Unskips `useUpgradeSecurityPackages` tests from [#112910](#112910)

Note: I added jest tests for the `useUpgradeSecurityPackages` changes, however didn't find a reasonable way to test the `prebuiltRulesPackageVersion` configuration addition via FTR's, so ended up testing that manually by running a local `package-registry` and serving up two different versions of the `security_detection_engine` package (`8.3.1`/`8.4.1`) and specifying 

> xpack.securitySolution.prebuiltRulesPackageVersion: '8.3.1'

in my `kibana.dev.yml` to try and install the previous version. This initially failed as fleet would say the package is `out-of-date`

<p align="center">
  <img width="700" src="https://user-images.githubusercontent.com/2946766/211948816-69860629-6db0-4007-8786-3b08f7312baf.png" />
</p>

Since there was a higher version with the same `kibana.version` requirement: `kibana.version: ^8.4.0`. Modifying this for the higher version to `^8.9.0` then allowed for the installation of the `8.3.1` as specified in the `prebuiltRulesPackageVersion` setting:

<p align="center">
  <img width="700" src="https://user-images.githubusercontent.com/2946766/211946889-030c2fdd-6c7d-4124-a1dc-003b54982311.png" />
</p>

<p align="center">
  <img width="700" src="https://user-images.githubusercontent.com/2946766/211948135-03163b0f-b1c2-435a-b91f-c3cbbe028053.png" />
</p>

As [mentioned](#148426 (comment)) by @xcrzx, I ended up adding `force:true` to the individual install request to get around this limitation and to have a better testing experience within Cypress.

Note II: When using the `prebuiltRulesPackageVersion` setting, since this is used for updates initiated from the client and not on kibana start like the `fleet_package.json` (added in #143839), you will have to uninstall the package that was installed on start-up for this to be successful. 

Note III: When wanting to run the Cypress tests against a specific package version, be sure to update the cypress FTR configuration [cf3a83f](cf3a83f).

### Checklist

Delete any items that are not applicable to this PR.

- [X] [Documentation](https://www.elastic.co/guide/en/kibana/master/development-documentation.html) was added for features that require explanation or tutorials
- [X] [Unit or functional tests](https://www.elastic.co/guide/en/kibana/master/development-tests.html) were updated or added to match the most common scenarios
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
failed-test A test failure on a tracked branch, potentially flaky-test Team: SecuritySolution Security Solutions Team working on SIEM, Endpoint, Timeline, Resolver, etc.
Projects
None yet
Development

Successfully merging a pull request may close this issue.

5 participants