Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Reporting/Screenshots] Do not fail the report if request is aborted #52344

Merged

Conversation

tsullivan
Copy link
Member

@tsullivan tsullivan commented Dec 5, 2019

Summary

Closes #52350

Release note: Fixed an issue in Reporting to stop the job from failing when there were aborted requests on the Kibana page.

Kibana applications have the ability now to abort network requests that have been fired, without waiting for a response, and this can happen during rendering. This seems to happen most of the time when Elasticsearch is finding search results with no cache (fresh restart of Elasticsearch).

These situations would cause Reporting jobs to fail, because we would fail a job when there were network errors detected.

This PR simply logs network errors as a warning, and assumes the reporting job still has a chance to complete successfully.

Here is the new warning log item:

server log [14:37:07.349] [warning][browser-driver][reporting] Request to [https://localhost/kbn/elasticsearch/kibana_sample_data_ecommerce/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1575581823467&timeout=30000ms] failed! [net::ERR_ABORTED]. This error will be ignored.

Here are full logs with the warning in context:

server    log   [14:36:55.761] [info][queue-job][reporting] Successfully queued job: k3t8qdn90c9bcbae0c4mnvnt
server    log   [14:36:56.999] [info][esqueue][queue-worker][reporting] k3t8n1mr0c9bcbae0ccabins - Job marked as claimed: /.reporting-2019.12.01/k3t8qdn90c9bcbae0c4mnvnt
server    log   [14:36:57.000] [info][esqueue][queue-worker][reporting] k3t8n1mr0c9bcbae0ccabins - Starting job
server    log   [14:36:57.583] [info][PNG][execute][k3t8qdn90c9bcbae0c4mnvnt][reporting] opening url https://localhost:443/kbn/app/kibana#/dashboard/722b74f0-b882-11e8-a6d9-e546fe2bba5f?_g=(refreshInterval%3A(pause%3A!f%2Cvalue%3A900000)%2Ctime%3A(from%3Anow-7d%2Cto%3Anow))&_a=(description%3A'Analyze%20mock%20eCommerce%20orders%20and%20revenue'%2Cfilters%3A!()%2CfullScreenMode%3A!f%2Coptions%3A(hidePanelTitles%3A!f%2CuseMargins%3A!t)%2Cpanels%3A!((embeddableConfig%3A(vis%3A(colors%3A('Men!'s%20Accessories'%3A%2382B5D8%2C'Men!'s%20Clothing'%3A%23F9BA8F%2C'Men!'s%20Shoes'%3A%23F29191%2C'Women!'s%20Accessories'%3A%23F4D598%2C'Women!'s%20Clothing'%3A%2370DBED%2C'Women!'s%20Shoes'%3A%23B7DBAB)))%2CgridData%3A(h%3A10%2Ci%3A'1'%2Cw%3A36%2Cx%3A12%2Cy%3A18)%2Cid%3A'37cc8650-b882-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'1'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(vis%3A(colors%3A(FEMALE%3A%236ED0E0%2CMALE%3A%23447EBC)%2ClegendOpen%3A!f))%2CgridData%3A(h%3A11%2Ci%3A'2'%2Cw%3A12%2Cx%3A12%2Cy%3A7)%2Cid%3Aed8436b0-b88b-11e8-a6d9-e546fe2bba5f%2CpanelIndex%3A'2'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A7%2Ci%3A'3'%2Cw%3A18%2Cx%3A0%2Cy%3A0)%2Cid%3A'09ffee60-b88c-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'3'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A7%2Ci%3A'4'%2Cw%3A30%2Cx%3A18%2Cy%3A0)%2Cid%3A'1c389590-b88d-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'4'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A11%2Ci%3A'5'%2Cw%3A48%2Cx%3A0%2Cy%3A28)%2Cid%3A'45e07720-b890-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'5'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A10%2Ci%3A'6'%2Cw%3A12%2Cx%3A0%2Cy%3A18)%2Cid%3A'10f1a240-b891-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'6'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A11%2Ci%3A'7'%2Cw%3A12%2Cx%3A0%2Cy%3A7)%2Cid%3Ab80e6540-b891-11e8-a6d9-e546fe2bba5f%2CpanelIndex%3A'7'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(vis%3A(colors%3A('0%20-%2050'%3A%23E24D42%2C'50%20-%2075'%3A%23EAB839%2C'75%20-%20100'%3A%237EB26D)%2CdefaultColors%3A('0%20-%2050'%3A'rgb(165%2C0%2C38)'%2C'50%20-%2075'%3A'rgb(255%2C255%2C190)'%2C'75%20-%20100'%3A'rgb(0%2C104%2C55)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A11%2Ci%3A'8'%2Cw%3A12%2Cx%3A24%2Cy%3A7)%2Cid%3A'4b3ec120-b892-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'8'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(vis%3A(colors%3A('0%20-%202'%3A%23E24D42%2C'2%20-%203'%3A%23F2C96D%2C'3%20-%204'%3A%239AC48A)%2CdefaultColors%3A('0%20-%202'%3A'rgb(165%2C0%2C38)'%2C'2%20-%203'%3A'rgb(255%2C255%2C190)'%2C'3%20-%204'%3A'rgb(0%2C104%2C55)')%2ClegendOpen%3A!f))%2CgridData%3A(h%3A11%2Ci%3A'9'%2Cw%3A12%2Cx%3A36%2Cy%3A7)%2Cid%3A'9ca7aa90-b892-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'9'%2Ctype%3Avisualization%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A18%2Ci%3A'10'%2Cw%3A48%2Cx%3A0%2Cy%3A54)%2Cid%3A'3ba638e0-b894-11e8-a6d9-e546fe2bba5f'%2CpanelIndex%3A'10'%2Ctype%3Asearch%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A(isLayerTOCOpen%3A!f%2CmapCenter%3A(lat%3A45.88578%2Clon%3A-15.07605%2Czoom%3A2.11)%2CopenTOCDetails%3A!())%2CgridData%3A(h%3A15%2Ci%3A'11'%2Cw%3A24%2Cx%3A0%2Cy%3A39)%2Cid%3A'2c9c1f60-1909-11e9-919b-ffe5949a18d2'%2CpanelIndex%3A'11'%2Ctype%3Amap%2Cversion%3A'8.0.0')%2C(embeddableConfig%3A()%2CgridData%3A(h%3A15%2Ci%3A'12'%2Cw%3A24%2Cx%3A24%2Cy%3A39)%2Cid%3Ab72dd430-bb4d-11e8-9c84-77068524bcab%2CpanelIndex%3A'12'%2Ctype%3Avisualization%2Cversion%3A'8.0.0'))%2Cquery%3A(language%3Akuery%2Cquery%3A'')%2CtimeRestore%3A!t%2Ctitle%3A'%5BeCommerce%5D%20Revenue%20Dashboard'%2CviewMode%3Aview)&forceNow=2019-12-05T21%3A36%3A55.700Z
server    log   [14:36:58.279] [error][browser-driver][headless-browser-console][reporting] Refused to execute inline script because it violates the following Content Security Policy directive: "script-src 'unsafe-eval' 'self'". Either the 'unsafe-inline' keyword, a hash ('sha256-SbBSU7MfZFnVMq4PuE/jbBz7pPIfXUTYDrdHl7Ckchc='), or a nonce ('nonce-...') is required to enable inline execution.

server    log   [14:37:04.751] [info][PNG][execute][k3t8qdn90c9bcbae0c4mnvnt][reporting] handled 94 page requests
server    log   [14:37:06.674] [info][PNG][execute][k3t8qdn90c9bcbae0c4mnvnt][reporting] found 12 rendered elements in the DOM
server    log   [14:37:07.348] [warning][browser-driver][reporting] Request to [https://localhost/kbn/elasticsearch/kibana_sample_data_ecommerce/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1575581823467&timeout=30000ms] failed! [net::ERR_ABORTED]. This error will be ignored.
server    log   [14:37:07.349] [warning][browser-driver][reporting] Request to [https://localhost/kbn/elasticsearch/kibana_sample_data_ecommerce/_search?rest_total_hits_as_int=true&ignore_unavailable=true&ignore_throttled=true&preference=1575581823467&timeout=30000ms] failed! [net::ERR_ABORTED]. This error will be ignored.
server    log   [14:37:13.829] [info][PNG][execute][k3t8qdn90c9bcbae0c4mnvnt][reporting] timeRange: Nov 28, 2019 @ 14:37:05.936 to Dec 5, 2019 @ 14:37:05.936
server    log   [14:37:13.834] [info][PNG][execute][k3t8qdn90c9bcbae0c4mnvnt][reporting] taking screenshots
server    log   [14:37:15.927] [info][PNG][execute][k3t8qdn90c9bcbae0c4mnvnt][reporting] screenshots taken: 1
server    log   [14:37:15.930] [info][esqueue][queue-worker][reporting] k3t8n1mr0c9bcbae0ccabins - Job execution completed successfully
server    log   [14:37:16.034] [error][browser-driver][reporting] error deleting user data directory at [/var/folders/2q/v3zwk__j5jbf0g_c2ls2pxdc0000gn/T/chromium-sNXx32]: [Error: Cannot delete files/directories outside the current working directory. Can be overridden with the `force` option.]
server    log   [14:37:16.442] [info][esqueue][queue-worker][reporting] k3t8n1mr0c9bcbae0ccabins - Job data saved successfully: /.reporting-2019.12.01/k3t8qdn90c9bcbae0c4mnvnt

I am not sure how this could be covered by testing.

Checklist

Use strikethroughs to remove checklist items you don't feel are applicable to this PR.

@elasticmachine
Copy link
Contributor

💚 Build Succeeded

@tsullivan tsullivan force-pushed the reporting/ignore-network-request-failures branch from 12c8bcc to d477a82 Compare December 6, 2019 20:23
@tsullivan tsullivan added (Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:skip Skip the PR/issue when compiling release notes review Team:Stack Services v7.6.0 v8.0.0 labels Dec 9, 2019
@elasticmachine
Copy link
Contributor

Pinging @elastic/kibana-stack-services (Team:Stack Services)

@tsullivan tsullivan requested a review from LeeDr December 9, 2019 17:38
@tsullivan
Copy link
Member Author

@elasticmachine merge upstream

tap(line => {
getBrowserLogger(page: Page): Rx.Observable<void> {
const consoleMessages$ = Rx.fromEvent<ConsoleMessage>(page, 'console').pipe(
map(line => {
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

switching from tap to map makes this an Observable of void

@LeeDr
Copy link

LeeDr commented Dec 10, 2019

@tsullivan I'm not sure I can actually test this correctly until it's merged and in a snapshot build. I'll try it locally and report my results but if it passes, I don't think it will be definitive.

Copy link
Member

@Bamieh Bamieh left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

@LeeDr
Copy link

LeeDr commented Dec 10, 2019

I'm struggling to get all the setup correct for my watcher/reporting test when running from this PR source. But I did get to the point of seeing the This error will be ignored. messages in the log.

@tsullivan tsullivan merged commit e71deb2 into elastic:master Dec 10, 2019
@tsullivan tsullivan deleted the reporting/ignore-network-request-failures branch December 10, 2019 16:24
tsullivan added a commit to tsullivan/kibana that referenced this pull request Dec 10, 2019
…lastic#52344)

* [Reporting/Screenshots] Do not fail the report if request is aborted

* take pageRequestFailed out of pageExit observable
@tsullivan
Copy link
Member Author

7.x/7.6.0: #52653

tsullivan added a commit that referenced this pull request Dec 10, 2019
…52344) (#52653)

* [Reporting/Screenshots] Do not fail the report if request is aborted

* take pageRequestFailed out of pageExit observable
jloleysens added a commit to jloleysens/kibana that referenced this pull request Dec 11, 2019
…le-sql-highlighting

* 'master' of github.com:elastic/kibana: (56 commits)
  Migrate url shortener service (elastic#50896)
  Re-enable datemath in from/to canvas timelion args (elastic#52159)
  [Logs + Metrics UI] Remove eslint exceptions (elastic#50979)
  [Logs + Metrics UI] Add missing headers in Logs & metrics (elastic#52405)
  [ML] API integration tests - initial tests for bucket span estimator (elastic#52636)
  [Watcher] New Platform (NP) Migration (elastic#50908)
  Decouple Authorization subsystem from Legacy API. (elastic#52638)
  [APM] Fix some warnings logged in APM tests (elastic#52487)
  [ui/public/utils] Delete unused base_object & find_by_param (elastic#52500)
  [ui/public/utils] Move items into ui/vis (elastic#52615)
  fix newlines in kbn-analytics build script
  Add top level examples folder and command to run, `--run-examples`. (elastic#52027)
  feat(NA): add trap for SIGINT in the git precommit hook (elastic#52662)
  [DOCS] Updtes description of elasticsearch.requestHeadersWhitelist (elastic#52675)
  [Telemetry/Pulse] Updates advanced settings text for usage data (elastic#52657)
  [SIEM][Detection Engine] Adds the default name space to the end of the signals index
  [Logs UI] Generalize ML module management (elastic#50662)
  Removing stateful saved object finder (elastic#52166)
  Shim oss telemetry (elastic#51168)
  [Reporting/Screenshots] Do not fail the report if request is aborted (elastic#52344)
  ...

# Conflicts:
#	src/legacy/core_plugins/console/public/legacy.ts
#	src/legacy/core_plugins/console/public/np_ready/application/models/legacy_core_editor/mode/elasticsearch_sql_highlight_rules.ts
#	src/legacy/core_plugins/console/public/np_ready/lib/autocomplete/components/full_request_component.ts
#	src/legacy/core_plugins/console/public/quarantined/src/sense_editor/row_parser.js
@tsullivan tsullivan added release_note:fix and removed release_note:skip Skip the PR/issue when compiling release notes labels Jan 3, 2020
@kibanamachine
Copy link
Contributor

💔 Build Failed

History

To update your PR or re-run it, just comment with:
@elasticmachine merge upstream

@elastic elastic deleted a comment from elasticmachine Jan 3, 2020
tsullivan added a commit to tsullivan/kibana that referenced this pull request Jan 13, 2020
…lastic#52344)

* [Reporting/Screenshots] Do not fail the report if request is aborted

* take pageRequestFailed out of pageExit observable
tsullivan added a commit that referenced this pull request Jan 13, 2020
…52344) (#54596)

* [Reporting/Screenshots] Do not fail the report if request is aborted

* take pageRequestFailed out of pageExit observable
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
(Deprecated) Feature:Reporting Use Reporting:Screenshot, Reporting:CSV, or Reporting:Framework instead release_note:fix review v7.5.2 v7.6.0 v8.0.0
Projects
None yet
Development

Successfully merging this pull request may close these issues.

Intermittent reporting failure with "Worker error"
5 participants