Skip to content

Commit

Permalink
[Reporting/Docs] Organize troubleshooting page and add CSV content (#…
Browse files Browse the repository at this point in the history
…182202)

## Summary

Closes elastic/platform-docs-team#309

The troubleshooting page for Kibana Reporting
([here](https://www.elastic.co/guide/en/kibana/current/reporting-troubleshooting.html))
page only covered screenshot-based reports (PNG, PDF). This PR updates
the guide to include advice for CSV reports as well.

---------

Co-authored-by: lcawl <lcawley@elastic.co>
  • Loading branch information
tsullivan and lcawl authored Jun 24, 2024
1 parent 4e86f5a commit 3c78092
Show file tree
Hide file tree
Showing 6 changed files with 260 additions and 142 deletions.
1 change: 1 addition & 0 deletions docs/settings/reporting-settings.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -245,6 +245,7 @@ Choose the API method used to page through data during CSV export. Valid options
[NOTE]
============
Each method has its own unique limitations which are important to understand.
* Scroll API: Search is limited to 500 shards at the very most. In cases where data shards are unavailable or time out, the export may return partial data.
* PIT API: Permissions to read data aliases alone will not work: the permissions are needed on the underlying indices or datastreams. In cases where data shards are unavailable or time out, the export will be empty rather than returning partial data.
============
Expand Down
2 changes: 2 additions & 0 deletions docs/user/reporting/index.asciidoc
Original file line number Diff line number Diff line change
Expand Up @@ -171,3 +171,5 @@ NOTE: *Public URL* is available only when anonymous access is configured and you

include::automating-report-generation.asciidoc[]
include::reporting-troubleshooting.asciidoc[]
include::reporting-csv-troubleshooting.asciidoc[leveloffset=+1]
include::reporting-pdf-troubleshooting.asciidoc[leveloffset=+1]
90 changes: 90 additions & 0 deletions docs/user/reporting/reporting-csv-troubleshooting.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,90 @@
[[reporting-troubleshooting-csv]]
== Troubleshooting CSV reports
++++
<titleabbrev>CSV</titleabbrev>
++++

The CSV export feature in Kibana makes queries to Elasticsearch and formats the results into CSV.
This feature offers a solution that attempts to provide the most benefit to the most use cases.
However, things could go wrong during export.
Elasticsearch can stop responding, repeated querying can take so long that authentication tokens can time
out, and the format of exported data can be too complex for spreadsheet applications to handle.
Such situations are outside of the control of Kibana.
If the use case becomes complex enough, it's recommended that you create scripts that query Elasticsearch directly, using a scripting language like Python and the public {es} APIs.

For more advice about common problems, refer to <<reporting-troubleshooting>>.

[NOTE]
============
It is recommended that you use CSV reports to export moderate amounts of data only.
The feature enables analysis of data in external tools, but it's not intended for bulk export or to backup {es} data.
If you need to export more than 250 MB of CSV, rather than increasing <<reporting-csv-settings,`xpack.reporting.csv.maxSizeBytes`>>, use
filters to create multiple smaller reports or extract the data you need directly from {es}.
The following deployment configurations may lead to failed report jobs or incomplete reports:
* Any shard needed for search is unavailable.
* Data is stored on slow storage tiers.
* Network latency between nodes is high.
* {ccs-cap} is used.
To export large amounts of data, use {es} APIs directly.
Check out the {ref}/point-in-time-api.html[Point in time API] or {ref}/sql-rest-format.html#_csv[SQL with CSV response data format].
============

[float]
[[reporting-troubleshooting-csv-configure-scan-api]]
=== Configuring CSV export to use the scroll API

The Kibana CSV export feature collects all of the data from Elasticsearch by using multiple requests to page
over all of the documents.
Internally, the feature uses the {ref}/point-in-time-api.html[Point in time API and
`search_after` parameters in the queries] to do so.
There are some limitations related to the point in time API:

1. Permissions to read data aliases alone will not work: the permissions are needed on the underlying indices or data streams.
2. In cases where data shards are unavailable or time out, the export will be empty rather than returning partial data.

Some users may benefit from using the {ref}/paginate-search-results.html#scroll-search-results[scroll API], an
alternative to paging through the data.
The behavior of this API does not have the limitations of point in time API, however it has its own limitations:

1. Search is limited to 500 shards at the very most.
2. In cases where the data shards are unavailable or time out, the export may return partial data.

If you prefer the internal implementation of CSV export to use the scroll API, you can configure this in
`kibana.yml`:

[source,yml]
-------------------------------------------
xpack.reporting.csv.scroll.strategy: scroll
-------------------------------------------

For more details about CSV export settings, go to <<reporting-csv-settings>>.

[float]
[[reporting-troubleshooting-csv-socket-hangup]]
=== Socket hangups

A "socket hangup" is a generic type of error meaning that a remote service (in this case Elasticsearch or a proxy in Cloud) closed the connection.
Kibana can't foresee when this might happen and can't force the remote service to keep the connection open.
To work around this situation, consider lowering the size of results that come back in each request or increase the amount of time the remote services will
allow to keep the request open.
For example:

[source,yml]
---------------------------------------
xpack.reporting.csv.scroll.size: 50
xpack.reporting.csv.scroll.duration: 2m
---------------------------------------

Such changes aren't guaranteed to solve the issue, but give the functionality a better
chance of working in this use case.
Unfortunately, lowering the scroll size will require more requests to Elasticsearch during export, which adds more time overhead, which could unintentionally create more instances of auth token expiration errors.

[float]
[[reporting-troubleshooting-csv-token-expired]]
=== Token expiration

To avoid token expirations, use a type of authentication that doesn't expire (such as Basic auth) or run the export using scripts that query Elasticsearch directly.
In a custom script, you have the ability to refresh the auth token as needed, such as once before each query.
136 changes: 136 additions & 0 deletions docs/user/reporting/reporting-pdf-troubleshooting.asciidoc
Original file line number Diff line number Diff line change
@@ -0,0 +1,136 @@
[[reporting-troubleshooting-pdf]]
== Troubleshooting PDF and PNG reports
++++
<titleabbrev>PDF/PNG</titleabbrev>
++++

For the most reliable configuration of PDF/PNG {report-features}, consider installing {kib} using <<docker,Docker>> or using <<set-up-on-cloud,Elastic Cloud>>.

For more advice about common problems, refer to <<reporting-troubleshooting>>.

[float]
[[reporting-diagnostics]]
=== Reporting diagnostics
Reporting comes with a built-in utility to try to automatically find common issues.
When {kib} is running, navigate to the *Report Listing* page, and click *Run reporting diagnostics*.
This will open up a diagnostic tool that checks various parts of the {kib} deployment and comes up with any relevant recommendations.

If the diagnostic information doesn't reveal the problem, you can troubleshoot further by starting the Kibana server with an environment variable for revealing additional debugging logs.
Refer to <<reporting-troubleshooting-puppeteer-debug-logs>>.

[float]
[[reporting-troubleshooting-nss-dependency]]
=== Network security service libraries

You must install Network Security Service (NSS) libraries for {report-features} to work.
Reporting using the Chromium browser relies on these libraries.
Install the appropriate nss package for your distribution.
Refer to <<install-reporting-packages>>.

[float]
[[reporting-troubleshooting-sandbox-dependency]]
=== Chromium sandbox requirements

Chromium uses sandboxing techniques that are built on top of operating system primitives.
The Linux sandbox depends on user namespaces, which were introduced with the 3.8 Linux kernel.
However, many distributions don't have user namespaces enabled by default or they require the CAP_SYS_ADMIN capability.
If the sandbox is not explicitly disabled in Kibana, either based on operating system detection or with the `xpack.screenshotting.browser.chromium.disableSandbox` setting, Chrome will try to enable the sandbox.
If it fails due to operating system or permissions restrictions, Chrome will crash during initialization.

Elastic recommends that you research the feasibility of enabling unprivileged user namespaces before disabling the sandbox.
An exception is if you are running Kibana in Docker because the container runs in a user namespace with the built-in seccomp/bpf filters.

[float]
[[reporting-troubleshooting-text-incorrect]]
=== Text rendered incorrectly in generated reports

If a report label is rendered as an empty rectangle, no system fonts are available.
Install at least one font package on the system.

If the report is missing certain Chinese, Japanese or Korean characters, ensure that a system font with those characters is installed.

[float]
[[reporting-troubleshooting-missing-data]]
=== Missing data in PDF report of data table visualization

There is currently a known limitation with the data table visualization that only the first page of data rows, which are the only data
visible on the screen, are shown in PDF reports.

[float]
[[reporting-troubleshooting-pdf-connection-refused]]
=== Connection refused errors

If PDF or PNG reports are not working due to a "connection refused" or "unable to connect" type of error, ensure that the `kibana.yml`
file uses the setting of `server.host: 0.0.0.0`.
Also verify that no firewall rules or other routing rules prevent local services from accessing this address.
Find out more at <<set-reporting-server-host>>.

[float]
[[reporting-troubleshooting-file-permissions]]
=== File permissions

Ensure that the `headless_shell` binary located in your Kibana data directory is owned by the user who is running Kibana, that the
user has the execute permission, and if applicable, that the filesystem is mounted with the `exec` option.

[NOTE]
--
The Chromium binary is located in the Kibana installation directory as `data/headless_shell-OS_TYPE/headless_shell`.
The full path is logged the first time Kibana starts when verbose logging is enabled.
--

[float]
[[reporting-troubleshooting-puppeteer-debug-logs]]
=== Puppeteer debug logs

The Chromium browser that {kib} launches on the server is driven by a NodeJS library for Chromium called Puppeteer.
The Puppeteer library has its own command-line method to generate its own debug logs, which can sometimes be helpful, particularly to figure out if a problem is caused by Kibana or Chromium.
Learn more https://github.com/GoogleChrome/puppeteer/blob/v1.19.0/README.md#debugging-tips[debugging tips].

Using Puppeteer's debug method when launching Kibana would look like:
```
env DEBUG="puppeteer:*" ./bin/kibana
```
The internal DevTools protocol traffic will be logged via the `debug` module under the `puppeteer` namespace.

The Puppeteer logs are very verbose and could possibly contain sensitive information.
Handle the generated output with care.

[float]
[[reporting-troubleshooting-system-requirements]]
=== System requirements

In Elastic Cloud, the {kib} instances that most configurations provide by default is for 1GB of RAM for the instance.
That is enough for {kib} {report-features} when the visualization or dashboard is relatively simple, such as a single pie chart or a dashboard with a few visualizations.
However, certain visualization types incur more load than others.
For example, a TSVB panel has a lot of network requests to render.

If the {kib} instance doesn't have enough memory to run the report, the report fails with an error such as `Error: Page crashed!`.
In this case, try increasing the memory for the {kib} instance to 2GB.

[float]
[[reporting-troubleshooting-maps-ems]]
=== Unable to connect to Elastic Maps Service

https://www.elastic.co/elastic-maps-service[{ems} ({ems-init})] is a service that hosts tile layers and vector shapes of administrative boundaries.
If a report contains a map with a missing basemap layer or administrative boundary, the {kib} server does not have access to {ems-init}.
Refer to <<maps-connect-to-ems>> for information about how to connect your {kib} server to {ems-init}.

[float]
[[reporting-manual-chromium-install]]
=== Manually install the Chromium browser for Darwin

Chromium is not embedded into {kib} for the Darwin (Mac OS) architecture.
When running {kib} on Darwin, {report-features} will download Chromium into the proper area of the {kib} installation path the first time the server starts.
If the server does not have access to the internet, you must download the Chromium browser and install it into the {kib} installation path.

1. Download the Chromium zip file:

** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac/901912/chrome-mac.zip[x64] systems
** For https://commondatastorage.googleapis.com/chromium-browser-snapshots/Mac_Arm/901913/chrome-mac.zip[ARM] systems

2. Copy the zip file into the holding area. Relative to the root directory of {kib}, the path is:

** `.chromium/x64` for x64 systems
** `.chromium/arm64` for ARM systems

When {kib} starts, it will automatically extract the browser from the zip file and is then ready for PNG and PDF reports.
Loading

0 comments on commit 3c78092

Please sign in to comment.