From c60a9cb9e4d074aaf0cb801075206c67081a0acd Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Thu, 23 Jun 2022 15:28:08 -0700 Subject: [PATCH 01/11] Restores legacy logging kibana configuration documentation with enhancements --- docs/setup/settings.asciidoc | 30 +++++++++++++++++++++--------- 1 file changed, 21 insertions(+), 9 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index b3da2059ff341..6c77c21047bf7 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -114,30 +114,42 @@ in this setting. `elasticsearch.logQueries`:: deprecated:[7.12.0,"This setting is no longer used and will be removed in Kibana 8.0."] -Instead, configure the `elasticsearch.query` logger. +Instead, configure the `elasticsearch.query` logger and set an approriate appender for the logger. + This is useful for seeing the query DSL generated by applications that currently do not have an inspector, for example Timelion and Monitoring. *Default: `false`* + -The following example shows a valid `elasticsearch.query` logger configuration: +The following example shows a full, valid `elasticsearch.query` logging configuration: + -[source,text] --- +[source,yaml] +-------------------------------------------------------------------------------- logging: appenders: - console_appender: - type: console + console_appender: <1> + type: console <2> layout: - type: pattern + type: pattern <3> highlight: true + pattern: "[%date][%level][%logger] %message" root: appenders: [default, console_appender] level: warn loggers: - name: elasticsearch.query - level: debug --- + level: debug <4> + appender: [console_appender] +-------------------------------------------------------------------------------- + +<1> Appender names are entirely custom with the restriction that appender names must be unique. Appenders may be used for multiple loggers. +<2> Console appender types will log to `stdout`. +<3> The `pattern` layout will use a default `[%date][%level][%logger] %message` configuration when not specified under the `pattern` key. +<4> Specifically logs the `elasticsearch.query` in debug mode, overriding the root logger configuration that will log other messages at a warn or error level. + +[NOTE] +============================================================================ +To get the most value from logging {es} queries, we recommend configuring the `http.server.response` logger as well. With both loggers configured, you will be able to see the responses from {es} on each request {kib} makes to {es}. +============================================================================ [[elasticsearch-pingTimeout]] `elasticsearch.pingTimeout`:: Time in milliseconds to wait for {es} to respond to pings. From 76af14fdf8f33d3c083bc7271b2a76ad2c0796a3 Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Thu, 23 Jun 2022 16:41:24 -0700 Subject: [PATCH 02/11] Adds legacy logging configuration settings back to setup/settings guide --- docs/setup/settings.asciidoc | 118 ++++++++++++++++++++++++++++++++--- 1 file changed, 111 insertions(+), 7 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 6c77c21047bf7..71f2b30f32cb4 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -131,14 +131,13 @@ logging: layout: type: pattern <3> highlight: true - pattern: "[%date][%level][%logger] %message" root: appenders: [default, console_appender] level: warn loggers: - - name: elasticsearch.query - level: debug <4> - appender: [console_appender] + - name: elasticsearch.query <4> + level: debug + appenders: [console_appender] -------------------------------------------------------------------------------- <1> Appender names are entirely custom with the restriction that appender names must be unique. Appenders may be used for multiple loggers. @@ -146,7 +145,7 @@ logging: <3> The `pattern` layout will use a default `[%date][%level][%logger] %message` configuration when not specified under the `pattern` key. <4> Specifically logs the `elasticsearch.query` in debug mode, overriding the root logger configuration that will log other messages at a warn or error level. -[NOTE] +[TIP] ============================================================================ To get the most value from logging {es} queries, we recommend configuring the `http.server.response` logger as well. With both loggers configured, you will be able to see the responses from {es} on each request {kib} makes to {es}. ============================================================================ @@ -308,9 +307,114 @@ This value must be a whole number greater than zero. *Default: `"1000"`* Maximum number of documents loaded by each shard to generate autocomplete suggestions. This value must be a whole number greater than zero. *Default: `"100000"`* -+ + NOTE: To reload the <>, send a SIGHUP signal to {kib}. +// deprecated logging configuration settings +[[logging-dest]] `logging.dest`:: + deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] + Instead, specify an appender `type`. Options are `console` for stdout or `file` for sending logs to file. + + + Enables you to specify a file where Kibana stores log output. *Default: `stdout`* + +`logging.json`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] + Instead, specify an appender `layout.type`. Options are `pattern` and `json`. + + + Logs output as JSON. When set to `true`, the logs are formatted as JSON + strings that include timestamp, log level, context, message text, and any other + metadata that may be associated with the log message. + When <> is set, and there is no interactive terminal ("TTY"),this setting defaults to `true`. *Default: `false`* + +`logging.quiet`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] + Instead, set the root logger `level` to `error` + + + Set the value of this setting to `true` to suppress all logging output other than error messages. *Default: `false`* + +`logging.rotate`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender + + +experimental[] Specifies the options for the logging rotate feature. +When not defined, all the sub options defaults would be applied. +The following example shows a valid logging rotate configuration: ++ +[source,text] +-- + logging.rotate: + enabled: true + everyBytes: 10485760 + keepFiles: 10 +-- + +`logging.rotate.enabled`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender + + +experimental[] Set the value of this setting to `true` to +enable log rotation. If you do not have a <> set that is different from `stdout` +that feature would not take any effect. *Default: `false`* + +`logging.rotate.everyBytes`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender + + +experimental[] The maximum size of a log file (that is `not an exact` limit). After the +limit is reached, a new log file is generated. The default size limit is 10485760 (10 MB) and +this option should be in the range of 1048576 (1 MB) to 1073741824 (1 GB). *Default: `10485760`* + +`logging.rotate.keepFiles`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender + + +experimental[] The number of most recent rotated log files to keep +on disk. Older files are deleted during log rotation. The default value is 7. The `logging.rotate.keepFiles` +option has to be in the range of 2 to 1024 files. *Default: `7`* + +`logging.rotate.pollingInterval`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender + + +experimental[] The number of milliseconds for the polling strategy in case +the <> is enabled. `logging.rotate.usePolling` must be in the 5000 to 3600000 millisecond range. *Default: `10000`* + +[[logging-rotate-usePolling]] `logging.rotate.usePolling`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender + + +experimental[] By default we try to understand the best way to monitoring +the log file and warning about it. Please be aware there are some systems where watch api is not accurate. In those cases, in order to get the feature working, +the `polling` method could be used enabling that option. *Default: `false`* + +`logging.silent`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] + Instead, set the root logger `level` to `off` + + +Set the value of this setting to `true` to +suppress all logging output. *Default: `false`* + +`logging.timezone`:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, use the {kibana-ref}/logging-service.html#pattern-layout[`pattern` appender layout] and specify a `%date` conversion pattern + + +Set to the canonical time zone ID +(for example, `America/Los_Angeles`) to log events using that time zone. +For possible values, refer to +https://en.wikipedia.org/wiki/List_of_tz_database_time_zones[database time zones]. *Default: `UTC`* + +[[logging-verbose]] `logging.verbose` {ess-icon}:: +deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] +Instead, set the root logger `level` to `debug` + + +Set to `true` to log all events, including system usage information and all +requests. *Default: `false`* + +[WARNING] +============================================================================ +While combining deprecated logging configuration with the current configuration, we strongly recommend switching over completely to avoid unforeseen consequences. If you use log rotation, combining deprecated with current configuration can lead to accidentally overwriting log files. +============================================================================ +// new logging configuration settings [[logging-root]] `logging.root`:: The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logging configuration is specified, `root` _must_ also be explicitly configured. @@ -336,7 +440,7 @@ Appender to use for logging records to *stdout*. By default, uses the `[%date][% `logging.appenders.file`:: Allows you to specify a fileName to send log records to on disk. To send <>, add the file appender to `root.appenders`. -`logging.appenders.rolling-file`:: +[[logging-appenders-rolling-file]]`logging.appenders.rolling-file`:: Similar to Log4j's `RollingFileAppender`, this appender will log into a file and rotate if following a rolling strategy when the configured policy triggers. There are currently two policies supported: `size-limit` and `time-interval`. + The size limit policy will perform a rollover when the log file reaches a maximum `size`. *Default 100mb* From 844b0fcbdf84b39329d3430055e65027cd980783 Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 11:34:17 -0700 Subject: [PATCH 03/11] Moves warning about mixing old and new logging styles --- docs/setup/settings.asciidoc | 17 +++++++++++------ 1 file changed, 11 insertions(+), 6 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 71f2b30f32cb4..73f55d82be35d 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -147,7 +147,8 @@ logging: [TIP] ============================================================================ -To get the most value from logging {es} queries, we recommend configuring the `http.server.response` logger as well. With both loggers configured, you will be able to see the responses from {es} on each request {kib} makes to {es}. +To get the most value from logging {es} queries, we recommend configuring the `http.server.response` logger as well. +With both loggers configured, {kib} will log the responses from {es} on each request {kib} makes to {es}. ============================================================================ [[elasticsearch-pingTimeout]] `elasticsearch.pingTimeout`:: @@ -308,12 +309,18 @@ Maximum number of documents loaded by each shard to generate autocomplete suggestions. This value must be a whole number greater than zero. *Default: `"100000"`* -NOTE: To reload the <>, send a SIGHUP signal to {kib}. +[WARNING] +============================================================================ +{kib} supports both deprecated and current logging systems being enabled at the same time. +We strongly suggest not mixing the configurations and rather switch over completely to avoid unforeseen ciscumstances. +For more information, refer to the <<`logging-configuration-migration`, logging configuration migration guide>> +============================================================================ // deprecated logging configuration settings [[logging-dest]] `logging.dest`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] Instead, specify an appender `type`. Options are `console` for stdout or `file` for sending logs to file. + Be aware that `logging.dest` might be set via CLI options in the `kibana.service` declaration. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. + Enables you to specify a file where Kibana stores log output. *Default: `stdout`* @@ -410,10 +417,8 @@ Instead, set the root logger `level` to `debug` Set to `true` to log all events, including system usage information and all requests. *Default: `false`* -[WARNING] -============================================================================ -While combining deprecated logging configuration with the current configuration, we strongly recommend switching over completely to avoid unforeseen consequences. If you use log rotation, combining deprecated with current configuration can lead to accidentally overwriting log files. -============================================================================ +NOTE: To reload the <>, send a SIGHUP signal to {kib}. + // new logging configuration settings [[logging-root]] `logging.root`:: The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logging configuration is specified, `root` _must_ also be explicitly configured. From 6ab901c2d7c0225affc2b294bda845b17b535ccc Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 11:43:31 -0700 Subject: [PATCH 04/11] Moves logging configuration migration to upgrade section --- docs/developer/architecture/index.asciidoc | 2 -- docs/setup/upgrade.asciidoc | 2 ++ .../upgrade}/logging-configuration-migration.asciidoc | 0 3 files changed, 2 insertions(+), 2 deletions(-) rename docs/{developer/architecture/core => setup/upgrade}/logging-configuration-migration.asciidoc (100%) diff --git a/docs/developer/architecture/index.asciidoc b/docs/developer/architecture/index.asciidoc index 90a0972d65f2f..774292f513f03 100644 --- a/docs/developer/architecture/index.asciidoc +++ b/docs/developer/architecture/index.asciidoc @@ -40,8 +40,6 @@ include::core/http-service.asciidoc[leveloffset=+1] include::core/logging-service.asciidoc[leveloffset=+1] -include::core/logging-configuration-migration.asciidoc[leveloffset=+1] - include::core/saved-objects-service.asciidoc[leveloffset=+1] include::core/uisettings-service.asciidoc[leveloffset=+1] diff --git a/docs/setup/upgrade.asciidoc b/docs/setup/upgrade.asciidoc index 3f395acd7320b..c84f3036c7bd8 100644 --- a/docs/setup/upgrade.asciidoc +++ b/docs/setup/upgrade.asciidoc @@ -134,3 +134,5 @@ add the setting manually include::upgrade/upgrade-standard.asciidoc[] include::upgrade/upgrade-migrations.asciidoc[] + +include::upgrade/logging-configuration-migration.asciidoc[] diff --git a/docs/developer/architecture/core/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc similarity index 100% rename from docs/developer/architecture/core/logging-configuration-migration.asciidoc rename to docs/setup/upgrade/logging-configuration-migration.asciidoc From 4558dd058d872e492e9f7b2b0d04a95a4d3804b0 Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 14:38:19 -0700 Subject: [PATCH 05/11] Updates debug level logging snippets --- docs/setup/settings.asciidoc | 25 +++++++++++++++---- .../logging-configuration-migration.asciidoc | 6 +++++ .../reporting-troubleshooting.asciidoc | 11 +++++++- 3 files changed, 36 insertions(+), 6 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 73f55d82be35d..a593a59108ca1 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -320,7 +320,6 @@ For more information, refer to the <<`logging-configuration-migration`, logging [[logging-dest]] `logging.dest`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] Instead, specify an appender `type`. Options are `console` for stdout or `file` for sending logs to file. - Be aware that `logging.dest` might be set via CLI options in the `kibana.service` declaration. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. + Enables you to specify a file where Kibana stores log output. *Default: `stdout`* @@ -421,10 +420,26 @@ NOTE: To reload the <>, send a SIGHUP signal // new logging configuration settings [[logging-root]] `logging.root`:: -The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logging configuration is specified, `root` _must_ also be explicitly configured. +The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logging configuration is specified, `root` _must_ also be explicitly configured with an appropriate appender. [[logging-root-level]] `logging.root.level` {ess-icon}:: -Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_ and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*. +Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_ and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*. ++ +The following example shows a full, valid `root` logging configuration that will log at the `debug` level: ++ +[source,yaml] +-------------------------------------------------------------------------------- +logging: + appenders: + console_appender: <1> + type: console <2> + layout: + type: pattern <3> + highlight: true + root: + appenders: [default, console_appender] + level: debug +-------------------------------------------------------------------------------- `logging.loggers.name`:: Specific logger instance. @@ -433,10 +448,10 @@ Specific logger instance. Level at which a log record should be shown. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. `logging.loggers.appenders`:: -Specific appender format to apply for a particular logger context. +Specific appender format to apply for a particular logger context. If not configured, falls back to the `root` logger appender if `logging.appenders`:: -{kibana-ref}/logging-service.html#logging-appenders[Appenders] define how and where log messages are displayed (eg. *stdout* or console) and stored (eg. file on the disk). +{kibana-ref}/logging-service.html#logging-appenders[Appenders] define how and where log messages are displayed (eg. *stdout* or console) and stored (eg. file on the disk). Appender configuration is required if any other logging configuration is specified. // TODO: add link to the advanced logging documentation. `logging.appenders.console`:: diff --git a/docs/setup/upgrade/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc index 19f10a881d5e8..781abdaccb01f 100644 --- a/docs/setup/upgrade/logging-configuration-migration.asciidoc +++ b/docs/setup/upgrade/logging-configuration-migration.asciidoc @@ -1,6 +1,12 @@ [[logging-configuration-migration]] == Logging configuration migration +{kib} supports mixed usage of the deprecated and current logging configuration styles, however, we recommend changing them all at once. + +If you use log rotation, be careful when mixing deprecated and current configuration styles as it could lead to unexpected behavior. For example, one system could apply the log file rotation while the other still writes to the original file. + +If you are using the service scripts (`kibana.service`), be aware that `logging.dest` might be set via CLI options in the `kibana.service` declaration. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. + Compatibility with the legacy logging system is assured until the end of the `v7` version. All log messages handled by `root` context are forwarded to the legacy logging service. If you re-write root appenders, make sure that it contains `default` appender to provide backward compatibility. diff --git a/docs/user/reporting/reporting-troubleshooting.asciidoc b/docs/user/reporting/reporting-troubleshooting.asciidoc index b42f083687812..f83369ae981ba 100644 --- a/docs/user/reporting/reporting-troubleshooting.asciidoc +++ b/docs/user/reporting/reporting-troubleshooting.asciidoc @@ -122,7 +122,16 @@ all, the full logs from Reporting will be the first place to look. In `kibana.ym [source,yaml] -------------------------------------------------------------------------------- -logging.root.level: all +logging: + appenders: + console_appender: <1> + type: console <2> + layout: + type: pattern <3> + highlight: true + root: + appenders: [default, console_appender] + level: debug -------------------------------------------------------------------------------- For more information about logging, see <>. From d27fcc62749adec715c3ee11cdab5d6e5b35684c Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 15:26:32 -0700 Subject: [PATCH 06/11] Adds an overview of how to prepare for logging in 8.0 --- docs/setup/settings.asciidoc | 3 +- .../logging-configuration-migration.asciidoc | 32 +++++++++++++------ 2 files changed, 24 insertions(+), 11 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index a593a59108ca1..484ff1f7e9969 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -420,7 +420,7 @@ NOTE: To reload the <>, send a SIGHUP signal // new logging configuration settings [[logging-root]] `logging.root`:: -The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logging configuration is specified, `root` _must_ also be explicitly configured with an appropriate appender. +The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logger is configured, the `root` logger _must_ also be explicitly configured with an appropriate appender. [[logging-root-level]] `logging.root.level` {ess-icon}:: Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_ and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*. @@ -452,7 +452,6 @@ Specific appender format to apply for a particular logger context. If not config `logging.appenders`:: {kibana-ref}/logging-service.html#logging-appenders[Appenders] define how and where log messages are displayed (eg. *stdout* or console) and stored (eg. file on the disk). Appender configuration is required if any other logging configuration is specified. -// TODO: add link to the advanced logging documentation. `logging.appenders.console`:: Appender to use for logging records to *stdout*. By default, uses the `[%date][%level][%logger] %message` **pattern** layout. To use a **json**, set the <>. diff --git a/docs/setup/upgrade/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc index 781abdaccb01f..f76d848f3a9c5 100644 --- a/docs/setup/upgrade/logging-configuration-migration.asciidoc +++ b/docs/setup/upgrade/logging-configuration-migration.asciidoc @@ -1,19 +1,33 @@ [[logging-configuration-migration]] == Logging configuration migration -{kib} supports mixed usage of the deprecated and current logging configuration styles, however, we recommend changing them all at once. - -If you use log rotation, be careful when mixing deprecated and current configuration styles as it could lead to unexpected behavior. For example, one system could apply the log file rotation while the other still writes to the original file. +[float] +=== Prepare for logging in 8.0 -If you are using the service scripts (`kibana.service`), be aware that `logging.dest` might be set via CLI options in the `kibana.service` declaration. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. +The entire logging system changed to offer a more consistent logging behavior with {es}. The current system differs from the more simplistic, deprecated system and allows you the freedom to customize what, how and where {kib} outputs log messages. While for logging based use cases, we recommend the <>, there are some important considerations: +[float] +==== Mixing deprecated and current configurations +{kib} supports mixed usage of the deprecated and current logging configuration styles, however, we recommend changing them all at once to reduce the risk of the systems interfering. + +[float] +==== Rotating log files +If you use log rotation, be careful when mixing deprecated and current configuration styles as it could lead to unexpected behavior. For example, one system could apply the log file rotation while the other still writes to the original file. Rather switch over to the current system completely. + +[float] +==== Service scripts +If you are using the service scripts (`kibana.service`), be aware that the deprecated `logging.dest` configuration might be set via CLI options in the `kibana.service` declaration. When `logging.dest` is set in the service scripts, {kib} will ignore any other log rotation configuration. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. + +[float] +==== Compatibility Compatibility with the legacy logging system is assured until the end of the `v7` version. -All log messages handled by `root` context are forwarded to the legacy logging service. If you re-write -root appenders, make sure that it contains `default` appender to provide backward compatibility. +All log messages handled by `root` context are forwarded to the legacy logging service. If you configure a custom appender for `root`, make sure to include the `default` appender in the appenders array to provide backward compatibility. + +[float] +==== Duplicate log messages +NOTE: When you switch to the new logging configuration, you will start seeing duplicate log entries in both legacy and current formats. The legacy format will be removed when the `default` appender is no longer required. -NOTE: When you switch to the new logging configuration, you will start seeing duplicate log entries in both formats. -These will be removed when the `default` appender is no longer required. If you define an appender for a logger, -the log messages aren't handled by the `root` logger anymore and are not forwarded to the legacy logging service. +To override this behavior for specific log messages, configure an appender for the logger. [[logging-pattern-format-old-and-new-example]] [options="header"] From bc646c114f120381c21d7dec61ad10844d4d74dc Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 15:43:45 -0700 Subject: [PATCH 07/11] adds full logging configuration to the cli-migration guide --- .../setup/upgrade/logging-configuration-migration.asciidoc | 7 +++++-- 1 file changed, 5 insertions(+), 2 deletions(-) diff --git a/docs/setup/upgrade/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc index f76d848f3a9c5..5162ebfaff3cf 100644 --- a/docs/setup/upgrade/logging-configuration-migration.asciidoc +++ b/docs/setup/upgrade/logging-configuration-migration.asciidoc @@ -74,7 +74,7 @@ To override this behavior for specific log messages, configure an appender for t As is the case for any of Kibana's config settings, you can specify your logging configuration via the CLI. For convenience, the `--verbose` and `--silent` flags exist as shortcuts and will continue to be supported beyond v7. -If you wish to override these flags, you can always do so by passing your preferred logging configuration directly to the CLI. For example, with the following configuration: +If you wish to override these flags, you can always do so by passing your preferred logging configuration directly to the CLI. For example, if {kib} has the following configuration in the `kibana.yml`, you can override the log level with CLI arguments, provided the full logging configuration is passed: [source,yaml] ---- @@ -85,9 +85,12 @@ logging: layout: type: pattern pattern: "[%date][%level] %message" + root: + appenders: [default, custom] + level: warn ---- -you can override the flags with: +You can override the flags with: [options="header"] |=== From e90fdbad78d9e77abd63f117a6dc2c81ee027c8f Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 16:08:55 -0700 Subject: [PATCH 08/11] Fixes typos --- docs/setup/settings.asciidoc | 27 +++++++++---------- .../logging-configuration-migration.asciidoc | 2 +- 2 files changed, 14 insertions(+), 15 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 484ff1f7e9969..16a019e29bb8e 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -114,7 +114,7 @@ in this setting. `elasticsearch.logQueries`:: deprecated:[7.12.0,"This setting is no longer used and will be removed in Kibana 8.0."] -Instead, configure the `elasticsearch.query` logger and set an approriate appender for the logger. +Instead, configure the `elasticsearch.query` logger and an appender for it. + This is useful for seeing the query DSL generated by applications that currently do not have an inspector, for example Timelion and Monitoring. @@ -311,8 +311,7 @@ suggestions. This value must be a whole number greater than zero. [WARNING] ============================================================================ -{kib} supports both deprecated and current logging systems being enabled at the same time. -We strongly suggest not mixing the configurations and rather switch over completely to avoid unforeseen ciscumstances. +While {kib} supports both deprecated and current logging systems being enabled at the same time, mixing the configurations is not recommended. To avoid unforeseen consequences, rather switch over to the current configuration completely. For more information, refer to the <<`logging-configuration-migration`, logging configuration migration guide>> ============================================================================ @@ -334,13 +333,13 @@ deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] `logging.quiet`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] - Instead, set the root logger `level` to `error` + Instead, set the root logger `level` to `error`. + Set the value of this setting to `true` to suppress all logging output other than error messages. *Default: `false`* `logging.rotate`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender. + experimental[] Specifies the options for the logging rotate feature. When not defined, all the sub options defaults would be applied. @@ -356,7 +355,7 @@ The following example shows a valid logging rotate configuration: `logging.rotate.enabled`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender. + experimental[] Set the value of this setting to `true` to enable log rotation. If you do not have a <> set that is different from `stdout` @@ -364,7 +363,7 @@ that feature would not take any effect. *Default: `false`* `logging.rotate.everyBytes`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender. + experimental[] The maximum size of a log file (that is `not an exact` limit). After the limit is reached, a new log file is generated. The default size limit is 10485760 (10 MB) and @@ -372,7 +371,7 @@ this option should be in the range of 1048576 (1 MB) to 1073741824 (1 GB). *Defa `logging.rotate.keepFiles`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender. + experimental[] The number of most recent rotated log files to keep on disk. Older files are deleted during log rotation. The default value is 7. The `logging.rotate.keepFiles` @@ -380,14 +379,14 @@ option has to be in the range of 2 to 1024 files. *Default: `7`* `logging.rotate.pollingInterval`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender. + experimental[] The number of milliseconds for the polling strategy in case the <> is enabled. `logging.rotate.usePolling` must be in the 5000 to 3600000 millisecond range. *Default: `10000`* [[logging-rotate-usePolling]] `logging.rotate.usePolling`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender +Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender. + experimental[] By default we try to understand the best way to monitoring the log file and warning about it. Please be aware there are some systems where watch api is not accurate. In those cases, in order to get the feature working, @@ -395,14 +394,14 @@ the `polling` method could be used enabling that option. *Default: `false`* `logging.silent`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] - Instead, set the root logger `level` to `off` + Instead, set the root logger `level` to `off`. + Set the value of this setting to `true` to suppress all logging output. *Default: `false`* `logging.timezone`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, use the {kibana-ref}/logging-service.html#pattern-layout[`pattern` appender layout] and specify a `%date` conversion pattern +Instead, use the {kibana-ref}/logging-service.html#pattern-layout[`pattern` appender layout] and specify a `%date` conversion pattern. + Set to the canonical time zone ID (for example, `America/Los_Angeles`) to log events using that time zone. @@ -411,7 +410,7 @@ https://en.wikipedia.org/wiki/List_of_tz_database_time_zones[database time zones [[logging-verbose]] `logging.verbose` {ess-icon}:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] -Instead, set the root logger `level` to `debug` +Instead, set the root logger `level` to `debug`. + Set to `true` to log all events, including system usage information and all requests. *Default: `false`* @@ -448,7 +447,7 @@ Specific logger instance. Level at which a log record should be shown. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. `logging.loggers.appenders`:: -Specific appender format to apply for a particular logger context. If not configured, falls back to the `root` logger appender if +Specific appender format to apply for a particular logger context. If not configured, falls back to the `root` logger appender. `logging.appenders`:: {kibana-ref}/logging-service.html#logging-appenders[Appenders] define how and where log messages are displayed (eg. *stdout* or console) and stored (eg. file on the disk). Appender configuration is required if any other logging configuration is specified. diff --git a/docs/setup/upgrade/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc index 5162ebfaff3cf..55ff4339e9cb6 100644 --- a/docs/setup/upgrade/logging-configuration-migration.asciidoc +++ b/docs/setup/upgrade/logging-configuration-migration.asciidoc @@ -16,7 +16,7 @@ If you use log rotation, be careful when mixing deprecated and current configura [float] ==== Service scripts -If you are using the service scripts (`kibana.service`), be aware that the deprecated `logging.dest` configuration might be set via CLI options in the `kibana.service` declaration. When `logging.dest` is set in the service scripts, {kib} will ignore any other log rotation configuration. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. +If you are using the service scripts (`kibana.service`), be aware that the deprecated `logging.dest` configuration might be set via CLI options in the `kibana.service` declaration. When `logging.dest` is set in the service scripts, {kib} will continue to write log messages to the "old" file. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. [float] ==== Compatibility From ac2da7cc514905ed031b21a479eee57f40e703e9 Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Fri, 24 Jun 2022 16:16:27 -0700 Subject: [PATCH 09/11] More small fixes --- docs/setup/settings.asciidoc | 6 +++--- docs/setup/upgrade/logging-configuration-migration.asciidoc | 3 +-- docs/user/reporting/reporting-troubleshooting.asciidoc | 6 +++--- 3 files changed, 7 insertions(+), 8 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 16a019e29bb8e..1be2e09e5f2e1 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -430,10 +430,10 @@ The following example shows a full, valid `root` logging configuration that will -------------------------------------------------------------------------------- logging: appenders: - console_appender: <1> - type: console <2> + console_appender: + type: console layout: - type: pattern <3> + type: pattern highlight: true root: appenders: [default, console_appender] diff --git a/docs/setup/upgrade/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc index 55ff4339e9cb6..393503d106a69 100644 --- a/docs/setup/upgrade/logging-configuration-migration.asciidoc +++ b/docs/setup/upgrade/logging-configuration-migration.asciidoc @@ -25,8 +25,7 @@ All log messages handled by `root` context are forwarded to the legacy logging s [float] ==== Duplicate log messages -NOTE: When you switch to the new logging configuration, you will start seeing duplicate log entries in both legacy and current formats. The legacy format will be removed when the `default` appender is no longer required. - +When you switch to the new logging configuration, you will start seeing duplicate log entries in both legacy and current formats. The legacy format will be removed when the `default` appender is no longer required. To override this behavior for specific log messages, configure an appender for the logger. [[logging-pattern-format-old-and-new-example]] diff --git a/docs/user/reporting/reporting-troubleshooting.asciidoc b/docs/user/reporting/reporting-troubleshooting.asciidoc index f83369ae981ba..e5cc3d396c5fd 100644 --- a/docs/user/reporting/reporting-troubleshooting.asciidoc +++ b/docs/user/reporting/reporting-troubleshooting.asciidoc @@ -124,10 +124,10 @@ all, the full logs from Reporting will be the first place to look. In `kibana.ym -------------------------------------------------------------------------------- logging: appenders: - console_appender: <1> - type: console <2> + console_appender: + type: console layout: - type: pattern <3> + type: pattern highlight: true root: appenders: [default, console_appender] From e1e688653fca3648640121e310ce16997b0efb21 Mon Sep 17 00:00:00 2001 From: Christiane Heiligers Date: Tue, 28 Jun 2022 07:02:54 -0700 Subject: [PATCH 10/11] Use template component --- docs/setup/settings.asciidoc | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 1be2e09e5f2e1..578a3b79c69fc 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -320,7 +320,7 @@ For more information, refer to the <<`logging-configuration-migration`, logging deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] Instead, specify an appender `type`. Options are `console` for stdout or `file` for sending logs to file. + - Enables you to specify a file where Kibana stores log output. *Default: `stdout`* + Enables you to specify a file where {kib} stores log output. *Default: `stdout`* `logging.json`:: deprecated:[7.13.0,This setting will be removed in Kibana 8.0.] From 3492e5eab157b0948abce34054306561a3552020 Mon Sep 17 00:00:00 2001 From: "Christiane (Tina) Heiligers" Date: Tue, 28 Jun 2022 16:02:25 -0700 Subject: [PATCH 11/11] Apply suggestions from code review Co-authored-by: Luke Elmers --- docs/setup/settings.asciidoc | 6 +++--- .../upgrade/logging-configuration-migration.asciidoc | 8 ++++---- 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/setup/settings.asciidoc b/docs/setup/settings.asciidoc index 578a3b79c69fc..c3494a77710fc 100644 --- a/docs/setup/settings.asciidoc +++ b/docs/setup/settings.asciidoc @@ -311,7 +311,7 @@ suggestions. This value must be a whole number greater than zero. [WARNING] ============================================================================ -While {kib} supports both deprecated and current logging systems being enabled at the same time, mixing the configurations is not recommended. To avoid unforeseen consequences, rather switch over to the current configuration completely. +While {kib} supports both deprecated and current logging systems being enabled at the same time, mixing the configurations is not recommended. To avoid unforeseen consequences, we recommend switching over to the current configuration completely. For more information, refer to the <<`logging-configuration-migration`, logging configuration migration guide>> ============================================================================ @@ -419,10 +419,10 @@ NOTE: To reload the <>, send a SIGHUP signal // new logging configuration settings [[logging-root]] `logging.root`:: -The `root` logger has is a <> and is pre-configured. The `root` logger logs at `info` level by default. If any other logger is configured, the `root` logger _must_ also be explicitly configured with an appropriate appender. +The `root` logger is a <> and is pre-configured. The `root` logger logs at `info` level by default. [[logging-root-level]] `logging.root.level` {ess-icon}:: -Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_ and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*. +Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_, and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*. + The following example shows a full, valid `root` logging configuration that will log at the `debug` level: + diff --git a/docs/setup/upgrade/logging-configuration-migration.asciidoc b/docs/setup/upgrade/logging-configuration-migration.asciidoc index 393503d106a69..34ff79f2ab150 100644 --- a/docs/setup/upgrade/logging-configuration-migration.asciidoc +++ b/docs/setup/upgrade/logging-configuration-migration.asciidoc @@ -4,19 +4,19 @@ [float] === Prepare for logging in 8.0 -The entire logging system changed to offer a more consistent logging behavior with {es}. The current system differs from the more simplistic, deprecated system and allows you the freedom to customize what, how and where {kib} outputs log messages. While for logging based use cases, we recommend the <>, there are some important considerations: +The entire logging system changed to offer a more consistent logging behavior with {es}, which uses the popular https://logging.apache.org/log4j/2.x[`log4j 2`] for managing logs. The current system differs from the more simplistic, deprecated system and allows you the freedom to customize what, how, and where {kib} outputs log messages. The best place to learn about the new logging system is our logging service guide. However, there are some important considerations when you are migrating from the old to the new system: [float] ==== Mixing deprecated and current configurations -{kib} supports mixed usage of the deprecated and current logging configuration styles, however, we recommend changing them all at once to reduce the risk of the systems interfering. +{kib} supports mixed usage of the deprecated and current logging configuration styles, however, to reduce the risk of confusion that may arise from conflicting configurations, we recommend changing over to the new system all at once. [float] ==== Rotating log files -If you use log rotation, be careful when mixing deprecated and current configuration styles as it could lead to unexpected behavior. For example, one system could apply the log file rotation while the other still writes to the original file. Rather switch over to the current system completely. +If you use log rotation, be careful when mixing deprecated and current configuration styles as it could lead to unexpected behavior. For example, one system could apply the log file rotation while the other still writes to the original file. Rather, switch over to the current system completely. [float] ==== Service scripts -If you are using the service scripts (`kibana.service`), be aware that the deprecated `logging.dest` configuration might be set via CLI options in the `kibana.service` declaration. When `logging.dest` is set in the service scripts, {kib} will continue to write log messages to the "old" file. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. +If you are using the systemd service scripts (`kibana.service`), be aware that the deprecated `logging.dest` configuration might be set via CLI options in the `kibana.service` declaration. When `logging.dest` is set in the service scripts, {kib} will continue to write log messages to the "old" file. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`. [float] ==== Compatibility