Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[@kbn/securitysolution-es-utils] remove transport API in favour of typed public API #113717

Merged
merged 5 commits into from
Oct 4, 2021
Merged

[@kbn/securitysolution-es-utils] remove transport API in favour of typed public API #113717

merged 5 commits into from
Oct 4, 2021

Conversation

mshustov
Copy link
Contributor

@mshustov mshustov commented Oct 4, 2021 •
edited
Loading

Summary

unblocks #108387
Prefer ES client API over transport API. It removes dependency on the transport protocol implementation and provides type safety.

@mshustov mshustov added chore v8.0.0 release_note:skip Skip the PR/issue when compiling release notes v7.16.0 labels Oct 4, 2021
mshustov
mshustov commented Oct 4, 2021
View reviewed changes
packages/kbn-securitysolution-es-utils/src/delete_policy/index.ts
})
).body;
// @ts-expect-error policy_id is required by mistake. fixed in the v8.0
(await esClient.ilm.deleteLifecycle({ policy })).body
Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

see https://github.com/elastic/elasticsearch-js/blob/main/src/api/api/ilm.ts

@mshustov mshustov changed the title remove transport API in favour of typed public API [@kbn/securitysolution-es-utils] remove transport API in favour of typed public API Oct 4, 2021
@kibanamachine
Copy link
Contributor

💛 Build succeeded, but was flaky

Test Failures

Kibana Pipeline / general / Performance Tests.x-pack/test/performance/tests/reporting_dashboard·ts.performance reporting dashbaord downloaded PDF has OK status

Link to Jenkins

Standard Out

Failed Tests Reporter:
  - Test has failed 3 times on tracked branches: https://github.com/elastic/kibana/issues/110470

[00:00:00]                         │
[00:00:00]                           └-: performance
[00:00:00]                             └-> "before all" hook in "performance"
[00:00:00]                             └-: reporting dashbaord
[00:00:00]                               └-> "before all" hook for "downloaded PDF has OK status"
[00:00:00]                               └-> "before all" hook for "downloaded PDF has OK status"
[00:00:00]                                 │ debg resolved import for x-pack/test/performance/kbn_archives/reporting_dashboard to /dev/shm/workspace/parallel/6/kibana/x-pack/test/performance/kbn_archives/reporting_dashboard.json
[00:00:00]                                 │ info importing 4 saved objects { space: undefined }
[00:00:00]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:00]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:00]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:00]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:00]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:00]                                 │ succ import success
[00:00:00]                                 │ info [x-pack/test/performance/es_archives/reporting_dashboard] Loading "mappings.json"
[00:00:00]                                 │ info [x-pack/test/performance/es_archives/reporting_dashboard] Loading "data.json.gz"
[00:00:00]                                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [foo] creating index, cause [api], templates [], shards [1]/[1]
[00:00:00]                                 │ info [x-pack/test/performance/es_archives/reporting_dashboard] Created index "foo"
[00:00:00]                                 │ debg [x-pack/test/performance/es_archives/reporting_dashboard] "foo" settings {"index":{"number_of_replicas":"1","number_of_shards":"1"}}
[00:00:02]                                 │ info [x-pack/test/performance/es_archives/reporting_dashboard] Indexed 10000 docs into "foo"
[00:00:02]                               └-> downloaded PDF has OK status
[00:00:02]                                 └-> "before each" hook: global before each for "downloaded PDF has OK status"
[00:00:02]                                 │ debg navigating to dashboards url: http://localhost:6161/app/dashboards
[00:00:02]                                 │ debg navigate to: http://localhost:6161/app/dashboards
[00:00:03]                                 │ debg browser[INFO] http://localhost:6161/login?next=%2Fapp%2Fdashboards%3F_t%3D1633349025404 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:03]                                 │
[00:00:03]                                 │ debg browser[INFO] http://localhost:6161/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:03]                                 │ debg ... sleep(700) start
[00:00:03]                                 │ debg ... sleep(700) end
[00:00:03]                                 │ debg returned from get, calling refresh
[00:00:05]                                 │ debg browser[INFO] http://localhost:6161/login?next=%2Fapp%2Fdashboards%3F_t%3D1633349025404 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:05]                                 │
[00:00:05]                                 │ debg browser[INFO] http://localhost:6161/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:05]                                 │ debg currentUrl = http://localhost:6161/login?next=%2Fapp%2Fdashboards%3F_t%3D1633349025404
[00:00:05]                                 │          appUrl = http://localhost:6161/app/dashboards
[00:00:05]                                 │ debg TestSubjects.find(kibanaChrome)
[00:00:05]                                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:06]                                 │ debg Found login page
[00:00:06]                                 │ debg TestSubjects.setValue(loginUsername, test_user)
[00:00:06]                                 │ debg TestSubjects.click(loginUsername)
[00:00:06]                                 │ debg Find.clickByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:06]                                 │ debg Find.findByCssSelector('[data-test-subj="loginUsername"]') with timeout=10000
[00:00:06]                                 │ERROR browser[SEVERE] http://localhost:6161/api/licensing/info - Failed to load resource: the server responded with a status of 401 (Unauthorized)
[00:00:06]                                 │ debg TestSubjects.setValue(loginPassword, changeme)
[00:00:06]                                 │ debg TestSubjects.click(loginPassword)
[00:00:06]                                 │ debg Find.clickByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:06]                                 │ debg Find.findByCssSelector('[data-test-subj="loginPassword"]') with timeout=10000
[00:00:06]                                 │ debg TestSubjects.click(loginSubmit)
[00:00:06]                                 │ debg Find.clickByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:06]                                 │ debg Find.findByCssSelector('[data-test-subj="loginSubmit"]') with timeout=10000
[00:00:06]                                 │ debg Find.waitForDeletedByCssSelector('.kibanaWelcomeLogo') with timeout=10000
[00:00:06]                                 │ proc [kibana]   log   [12:03:49.040] [info][plugins][routes][security] Logging in with provider "basic" (basic)
[00:00:07]                                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"]') with timeout=60000
[00:00:07]                                 │ debg Find.findByCssSelector('[data-test-subj="kibanaChrome"] nav:not(.ng-hide)') with timeout=60000
[00:00:09]                                 │ debg browser[INFO] http://localhost:6161/app/dashboards?_t=1633349025404 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:09]                                 │
[00:00:09]                                 │ debg browser[INFO] http://localhost:6161/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:10]                                 │ debg browser[INFO] http://localhost:6161/app/dashboards?_t=1633349032013 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:10]                                 │
[00:00:10]                                 │ debg browser[INFO] http://localhost:6161/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:10]                                 │ debg Finished login process currentUrl = http://localhost:6161/app/dashboards
[00:00:10]                                 │ debg ... sleep(501) start
[00:00:11]                                 │ debg ... sleep(501) end
[00:00:11]                                 │ debg in navigateTo url = http://localhost:6161/app/dashboards
[00:00:11]                                 │ debg Waiting up to 20000ms for dashboard landing page...
[00:00:11]                                 │ debg onDashboardLandingPage
[00:00:11]                                 │ debg TestSubjects.exists(dashboardLandingPage)
[00:00:11]                                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=5000
[00:00:12]                                 │ debg Load Saved Dashboard dashboard
[00:00:12]                                 │ debg gotoDashboardLandingPage
[00:00:12]                                 │ debg onDashboardLandingPage
[00:00:12]                                 │ debg TestSubjects.exists(dashboardLandingPage)
[00:00:12]                                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=5000
[00:00:12]                                 │ debg searchForItemWithName: dashboard
[00:00:12]                                 │ debg TestSubjects.find(tableListSearchBox)
[00:00:12]                                 │ debg Find.findByCssSelector('[data-test-subj="tableListSearchBox"]') with timeout=10000
[00:00:12]                                 │ debg isGlobalLoadingIndicatorVisible
[00:00:12]                                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:00:12]                                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:00:12]                                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:00:12]                                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:00:14]                                 │ debg TestSubjects.click(dashboardListingTitleLink-dashboard)
[00:00:14]                                 │ debg Find.clickByCssSelector('[data-test-subj="dashboardListingTitleLink-dashboard"]') with timeout=10000
[00:00:14]                                 │ debg Find.findByCssSelector('[data-test-subj="dashboardListingTitleLink-dashboard"]') with timeout=10000
[00:00:14]                                 │ debg isGlobalLoadingIndicatorVisible
[00:00:14]                                 │ debg TestSubjects.exists(globalLoadingIndicator)
[00:00:14]                                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="globalLoadingIndicator"]') with timeout=1500
[00:00:14]                                 │ debg browser[INFO] http://localhost:6161/app/dashboards#/view/37b49c50-2dc6-11eb-8af3-cb3aa84dbabd?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now)) 281 Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-P5polb1UreUSOe5V/Pv7tc+yeZuJXiOi/3fqhGsU7BE='), or a nonce ('nonce-...') is required to enable inline execution.
[00:00:14]                                 │
[00:00:14]                                 │ debg browser[INFO] http://localhost:6161/bootstrap.js 41:19 "^ A single error about an inline script not firing due to content security policy is expected!"
[00:00:15]                                 │ debg --- retry.tryForTime error: [data-test-subj="globalLoadingIndicator"] is not displayed
[00:00:16]                                 │ debg TestSubjects.exists(globalLoadingIndicator-hidden)
[00:00:16]                                 │ debg Find.existsByCssSelector('[data-test-subj="globalLoadingIndicator-hidden"]') with timeout=100000
[00:00:16]                                 │ debg TestSubjects.missingOrFail(dashboardLandingPage)
[00:00:16]                                 │ debg Find.waitForDeletedByCssSelector('[data-test-subj="dashboardLandingPage"]') with timeout=10000
[00:00:17]                                 │ debg openPdfReportingPanel
[00:00:17]                                 │ debg openShareMenuItem title:PDF Reports
[00:00:17]                                 │ debg TestSubjects.exists(shareContextMenu)
[00:00:17]                                 │ debg Find.existsByDisplayedByCssSelector('[data-test-subj="shareContextMenu"]') with timeout=2500
[00:00:17]                                 │ info [o.e.c.m.MetadataCreateIndexService] [node-01] [.async-search] creating index, cause [auto(bulk api)], templates [], shards [1]/[0]
[00:00:19]                                 │ debg --- retry.tryForTime error: [data-test-subj="shareContextMenu"] is not displayed
[00:00:20]                                 │ debg TestSubjects.click(shareTopNavButton)
[00:00:20]                                 │ debg Find.clickByCssSelector('[data-test-subj="shareTopNavButton"]') with timeout=10000
[00:00:20]                                 │ debg Find.findByCssSelector('[data-test-subj="shareTopNavButton"]') with timeout=10000
[00:00:20]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:21]                                 │ debg Find.findByCssSelector('div.euiContextMenuPanel') with timeout=10000
[00:00:21]                                 │ debg TestSubjects.click(sharePanel-PDFReports)
[00:00:21]                                 │ debg Find.clickByCssSelector('[data-test-subj="sharePanel-PDFReports"]') with timeout=10000
[00:00:21]                                 │ debg Find.findByCssSelector('[data-test-subj="sharePanel-PDFReports"]') with timeout=10000
[00:00:22]                                 │ debg Find.waitForElementStale with timeout=10000
[00:00:23]                                 │ debg TestSubjects.click(generateReportButton)
[00:00:23]                                 │ debg Find.clickByCssSelector('[data-test-subj="generateReportButton"]') with timeout=10000
[00:00:23]                                 │ debg Find.findByCssSelector('[data-test-subj="generateReportButton"]') with timeout=10000
[00:00:24]                                 │ info [o.e.c.m.MetadataMappingService] [node-01] [.kibana_8.0.0_001/UGfC0QnrRdW3GDdwvqXSNQ] update_mapping [_doc]
[00:00:24]                                 │ debg getReportURL
[00:00:24]                                 │ debg TestSubjects.getAttribute(downloadCompletedReportButton, href, tryTimeout=120000, findTimeout=60000)
[00:00:24]                                 │ debg TestSubjects.find(downloadCompletedReportButton)
[00:00:24]                                 │ debg Find.findByCssSelector('[data-test-subj="downloadCompletedReportButton"]') with timeout=60000
[00:01:25]                                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
[00:01:25]                                 │      Wait timed out after 61206ms
[00:01:26]                                 │ debg TestSubjects.find(downloadCompletedReportButton)
[00:01:26]                                 │ debg Find.findByCssSelector('[data-test-subj="downloadCompletedReportButton"]') with timeout=60000
[00:02:27]                                 │ debg --- retry.tryForTime error: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
[00:02:27]                                 │      Wait timed out after 61173ms
[00:02:27]                                 │ info Taking screenshot "/dev/shm/workspace/parallel/6/kibana/x-pack/test/functional/screenshots/failure/performance reporting dashbaord downloaded PDF has OK status.png"
[00:02:28]                                 │ info Current URL is: http://localhost:6161/app/dashboards#/view/37b49c50-2dc6-11eb-8af3-cb3aa84dbabd?_g=(filters:!(),refreshInterval:(pause:!t,value:0),time:(from:now-15m,to:now))
[00:02:28]                                 │ info Saving page source to: /dev/shm/workspace/parallel/6/kibana/x-pack/test/performance/failure_debug/html/performance reporting dashbaord downloaded PDF has OK status.html
[00:02:28]                                 └- ✖ fail: performance reporting dashbaord downloaded PDF has OK status
[00:02:28]                                 │      Error: retry.tryForTime timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
[00:02:28]                                 │ Wait timed out after 61173ms
[00:02:28]                                 │     at /dev/shm/workspace/parallel/6/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
[00:02:28]                                 │     at runMicrotasks (<anonymous>)
[00:02:28]                                 │     at processTicksAndRejections (internal/process/task_queues.js:95:5)
[00:02:28]                                 │       at onFailure (/dev/shm/workspace/parallel/6/kibana/test/common/services/retry/retry_for_success.ts:17:9)
[00:02:28]                                 │       at retryForSuccess (/dev/shm/workspace/parallel/6/kibana/test/common/services/retry/retry_for_success.ts:57:13)
[00:02:28]                                 │       at RetryService.tryForTime (/dev/shm/workspace/parallel/6/kibana/test/common/services/retry/retry.ts:22:12)
[00:02:28]                                 │       at TestSubjects.getAttribute (/dev/shm/workspace/parallel/6/kibana/test/functional/services/common/test_subjects.ts:190:12)
[00:02:28]                                 │       at ReportingPageObject.getReportURL (test/functional/page_objects/reporting_page.ts:36:17)
[00:02:28]                                 │       at Context.<anonymous> (test/performance/tests/reporting_dashboard.ts:50:7)
[00:02:28]                                 │       at Object.apply (/dev/shm/workspace/parallel/6/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)
[00:02:28]                                 │ 
[00:02:28]                                 │

Stack Trace

Error: retry.tryForTime timeout: TimeoutError: Waiting for element to be located By(css selector, [data-test-subj="downloadCompletedReportButton"])
Wait timed out after 61173ms
    at /dev/shm/workspace/parallel/6/kibana/node_modules/selenium-webdriver/lib/webdriver.js:842:17
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:95:5)
    at onFailure (/dev/shm/workspace/parallel/6/kibana/test/common/services/retry/retry_for_success.ts:17:9)
    at retryForSuccess (/dev/shm/workspace/parallel/6/kibana/test/common/services/retry/retry_for_success.ts:57:13)
    at RetryService.tryForTime (/dev/shm/workspace/parallel/6/kibana/test/common/services/retry/retry.ts:22:12)
    at TestSubjects.getAttribute (/dev/shm/workspace/parallel/6/kibana/test/functional/services/common/test_subjects.ts:190:12)
    at ReportingPageObject.getReportURL (test/functional/page_objects/reporting_page.ts:36:17)
    at Context.<anonymous> (test/performance/tests/reporting_dashboard.ts:50:7)
    at Object.apply (/dev/shm/workspace/parallel/6/kibana/node_modules/@kbn/test/target_node/functional_test_runner/lib/mocha/wrap_function.js:87:16)

Metrics [docs]

✅ unchanged

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@mshustov mshustov marked this pull request as ready for review October 4, 2021 13:11
@mshustov mshustov requested a review from a team as a code owner October 4, 2021 13:11
FrankHassanabad
FrankHassanabad approved these changes Oct 4, 2021
View reviewed changes
Copy link
Contributor

@FrankHassanabad FrankHassanabad left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM,

  • Looked it over carefully to ensure that the PR removes the transport.

Very interesting and good PR. Must be tricky to help other teams like this, so much appreciated 👍

FrankHassanabad
FrankHassanabad reviewed Oct 4, 2021
View reviewed changes
packages/kbn-securitysolution-es-utils/src/create_boostrap_index/index.ts
@@ -6,7 +6,7 @@
* Side Public License, v 1.
*/

import { ElasticsearchClient } from '../elasticsearch_client';
import type { ElasticsearchClient } from '../elasticsearch_client';
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 Thanks for the additional import type here.

@mshustov mshustov merged commit 1f60a16 into master Oct 4, 2021
@mshustov mshustov deleted the security-solution-no-transport branch October 4, 2021 18:42
@mshustov mshustov added the auto-backport Deprecated - use backport:version if exact versions are needed label Oct 4, 2021
kibanamachine pushed a commit to kibanamachine/kibana that referenced this pull request Oct 4, 2021
@mshustov @kibanamachine
[@kbn/securitysolution-es-utils] remove transport API in favour of ty…
57cd887 
…ped public API (elastic#113717)

* remove transport API in favour of typed public API

* put elasticsearch_client back

* fix create index call

* fix setpolicy

* fix unit tests in SecuritySolution
@kibanamachine kibanamachine mentioned this pull request Oct 4, 2021
Merged
@kibanamachine
Copy link
Contributor

💚 Backport successful

Status Branch Result
✅ 7.x

This backport PR will be merged automatically after passing CI.

kibanamachine added a commit that referenced this pull request Oct 4, 2021
@kibanamachine @mshustov
[@kbn/securitysolution-es-utils] remove transport API in favour of ty…
9f95a44 
…ped public API (#113717) (#113831)

* remove transport API in favour of typed public API

* put elasticsearch_client back

* fix create index call

* fix setpolicy

* fix unit tests in SecuritySolution

Co-authored-by: Mikhail Shustov <restrry@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@FrankHassanabad FrankHassanabad FrankHassanabad approved these changes

Assignees
No one assigned
Labels
auto-backport Deprecated - use backport:version if exact versions are needed chore release_note:skip Skip the PR/issue when compiling release notes v7.16.0 v8.0.0
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mshustov @kibanamachine @FrankHassanabad