Logging: Restore and enhance legacy logging docs

docs/developer/architecture/index.asciidoc

@@ -40,8 +40,6 @@ include::core/http-service.asciidoc[leveloffset=+1]
 
 include::core/logging-service.asciidoc[leveloffset=+1]
 
-include::core/logging-configuration-migration.asciidoc[leveloffset=+1]
-
 include::core/saved-objects-service.asciidoc[leveloffset=+1]
 
 include::core/uisettings-service.asciidoc[leveloffset=+1]

docs/setup/settings.asciidoc

@@ -114,30 +114,42 @@ in this setting.
 
 `elasticsearch.logQueries`::
 deprecated:[7.12.0,"This setting is no longer used and will be removed in Kibana 8.0."]
-Instead, configure the `elasticsearch.query` logger.
+Instead, configure the `elasticsearch.query` logger and an appender for it.
 +
 This is useful for seeing the query DSL generated by applications that
 currently do not have an inspector, for example Timelion and Monitoring.
 *Default: `false`*
 +
-The following example shows a valid `elasticsearch.query` logger configuration:
+The following example shows a full, valid `elasticsearch.query` logging configuration:
 +
-[source,text]
---
+[source,yaml]
+--------------------------------------------------------------------------------
 logging:
   appenders:
-    console_appender:
-      type: console
+    console_appender: <1>
+      type: console <2>
       layout:
-        type: pattern
+        type: pattern <3>
         highlight: true
   root:
     appenders: [default, console_appender]
     level: warn
   loggers:
-    - name: elasticsearch.query
+    - name: elasticsearch.query <4>
       level: debug
---
+      appenders: [console_appender]
+--------------------------------------------------------------------------------
+
+<1> Appender names are entirely custom with the restriction that appender names must be unique. Appenders may be used for multiple loggers.
+<2> Console appender types will log to `stdout`.
+<3> The `pattern` layout will use a default `[%date][%level][%logger] %message` configuration when not specified under the `pattern` key. 
+<4> Specifically logs the `elasticsearch.query` in debug mode, overriding the root logger configuration that will log other messages at a warn or error level.
+
+[TIP]
+============================================================================
+To get the most value from logging {es} queries, we recommend configuring the `http.server.response` logger as well. 
+With both loggers configured, {kib} will log the responses from {es} on each request {kib} makes to {es}.
+============================================================================
 
 [[elasticsearch-pingTimeout]] `elasticsearch.pingTimeout`::
 Time in milliseconds to wait for {es} to respond to pings.
@@ -296,14 +308,137 @@ This value must be a whole number greater than zero. *Default: `"1000"`*
 Maximum number of documents loaded by each shard to generate autocomplete
 suggestions. This value must be a whole number greater than zero.
 *Default: `"100000"`*
+
+[WARNING]
+============================================================================
+While {kib} supports both deprecated and current logging systems being enabled at the same time, mixing the configurations is not recommended. To avoid unforeseen consequences, rather switch over to the current configuration completely.
+For more information, refer to the <<`logging-configuration-migration`, logging configuration migration guide>>
+============================================================================
+
+// deprecated logging configuration settings
+[[logging-dest]] `logging.dest`::
+ deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+ Instead, specify an appender `type`. Options are `console` for stdout or `file` for sending logs to file.
+ +
+ Enables you to specify a file where Kibana stores log output. *Default: `stdout`*
+
+`logging.json`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+ Instead, specify an appender `layout.type`. Options are `pattern` and `json`.
+ +
+ Logs output as JSON. When set to `true`, the logs are formatted as JSON
+ strings that include timestamp, log level, context, message text, and any other
+ metadata that may be associated with the log message.
+ When <<logging-dest, `logging.dest.stdout`>> is set, and there is no interactive terminal ("TTY"),this setting defaults to `true`. *Default: `false`*
+
+`logging.quiet`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+ Instead, set the root logger `level` to `error`.
+ +
+ Set the value of this setting to `true` to suppress all logging output other than error messages. *Default: `false`*
+
+`logging.rotate`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender.
+ +
+experimental[] Specifies the options for the logging rotate feature.
+When not defined, all the sub options defaults would be applied.
+The following example shows a valid logging rotate configuration:
 +
+[source,text]
+--
+ logging.rotate:
+   enabled: true
+   everyBytes: 10485760
+   keepFiles: 10
+--
+
+`logging.rotate.enabled`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender.
+ +
+experimental[] Set the value of this setting to `true` to
+enable log rotation. If you do not have a <<logging-dest, `logging.dest`>> set that is different from `stdout`
+that feature would not take any effect. *Default: `false`*
+
+`logging.rotate.everyBytes`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender.
+ +
+experimental[] The maximum size of a log file (that is `not an exact` limit). After the
+limit is reached, a new log file is generated. The default size limit is 10485760 (10 MB) and
+this option should be in the range of 1048576 (1 MB) to 1073741824 (1 GB). *Default: `10485760`*
+
+`logging.rotate.keepFiles`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender.
+ +
+experimental[] The number of most recent rotated log files to keep
+on disk. Older files are deleted during log rotation. The default value is 7. The `logging.rotate.keepFiles`
+option has to be in the range of 2 to 1024 files. *Default: `7`*
+
+`logging.rotate.pollingInterval`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender.
+ +
+experimental[] The number of milliseconds for the polling strategy in case
+the <<logging-rotate-usePolling, `logging.rotate.usePolling`>> is enabled. `logging.rotate.usePolling` must be in the 5000 to 3600000 millisecond range. *Default: `10000`*
+
+[[logging-rotate-usePolling]] `logging.rotate.usePolling`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the <<`rolling-file`, logging.appenders.rolling.file>> appender.
+ +
+experimental[] By default we try to understand the best way to monitoring
+the log file and warning about it. Please be aware there are some systems where watch api is not accurate. In those cases, in order to get the feature working,
+the `polling` method could be used enabling that option. *Default: `false`*
+
+`logging.silent`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+ Instead, set the root logger `level` to `off`.
+ +
+Set the value of this setting to `true` to
+suppress all logging output. *Default: `false`*
+
+`logging.timezone`::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, use the {kibana-ref}/logging-service.html#pattern-layout[`pattern` appender layout] and specify a `%date` conversion pattern.
+ +
+Set to the canonical time zone ID
+(for example, `America/Los_Angeles`) to log events using that time zone.
+For possible values, refer to
+https://en.wikipedia.org/wiki/List_of_tz_database_time_zones[database time zones]. *Default: `UTC`*
+
+[[logging-verbose]] `logging.verbose` {ess-icon}::
+deprecated:[7.13.0,This setting will be removed in Kibana 8.0.]
+Instead, set the root logger `level` to `debug`.
+ +
+Set to `true` to log all events, including system usage information and all
+requests. *Default: `false`*
+
 NOTE: To reload the <<logging-settings, logging settings>>, send a SIGHUP signal to {kib}.
 
+// new logging configuration settings
 [[logging-root]] `logging.root`::
-The `root` logger has is a <<dedicated-loggers, dedicated logger>> and is pre-configured. The `root` logger logs at `info` level by default. If any other logging configuration is specified, `root` _must_ also be explicitly configured.
+The `root` logger has is a <<dedicated-loggers, dedicated logger>> and is pre-configured. The `root` logger logs at `info` level by default. If any other logger is configured, the `root` logger _must_ also be explicitly configured with an appropriate appender. 
 
 [[logging-root-level]] `logging.root.level` {ess-icon}::
-Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_ and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <<change-overall-log-level,change the overall log level>>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*.
+Level at which a log record should be logged. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_. Levels are ordered from _all_ (highest) to _off_ and a log record will be logged it its level is higher than or equal to the level of its logger, otherwise the log record is ignored. Use this value to <<change-overall-log-level,change the overall log level>>. Set to `all` to log all events, including system usage information and all requests. Set to `off` to silence all logs. *Default: `info`*. 
++
+The following example shows a full, valid `root` logging configuration that will log at the `debug` level:
++
+[source,yaml]
+--------------------------------------------------------------------------------
+logging:
+  appenders:
+    console_appender:
+      type: console
+      layout:
+        type: pattern
+        highlight: true
+  root:
+    appenders: [default, console_appender]
+    level: debug
+--------------------------------------------------------------------------------
 
 `logging.loggers.name`::
 Specific logger instance.
@@ -312,19 +447,18 @@ Specific logger instance.
 Level at which a log record should be shown. Supported levels are: _all_, _fatal_, _error_, _warn_, _info_, _debug_, _trace_, _off_.
 
 `logging.loggers.appenders`::
-Specific appender format to apply for a particular logger context.
+Specific appender format to apply for a particular logger context. If not configured, falls back to the `root` logger appender.
 
 `logging.appenders`::
-{kibana-ref}/logging-service.html#logging-appenders[Appenders] define how and where log messages are displayed (eg. *stdout* or console) and stored (eg. file on the disk).
-// TODO: add link to the advanced logging documentation.
+{kibana-ref}/logging-service.html#logging-appenders[Appenders] define how and where log messages are displayed (eg. *stdout* or console) and stored (eg. file on the disk). Appender configuration is required if any other logging configuration is specified.
 
 `logging.appenders.console`::
 Appender to use for logging records to *stdout*. By default, uses the `[%date][%level][%logger] %message` **pattern** layout. To use a **json**, set the <<log-in-json-ECS-example,layout type to `json`>>.
 
 `logging.appenders.file`::
 Allows you to specify a fileName to send log records to on disk. To send <<log-to-file-example,all log records to file>>, add the file appender to `root.appenders`.
 
-`logging.appenders.rolling-file`::
+[[logging-appenders-rolling-file]]`logging.appenders.rolling-file`::
 Similar to Log4j's `RollingFileAppender`, this appender will log into a file and rotate if following a rolling strategy when the configured policy triggers. There are currently two policies supported: `size-limit` and `time-interval`.
 +
 The size limit policy will perform a rollover when the log file reaches a maximum `size`. *Default 100mb*

docs/setup/upgrade.asciidoc

@@ -134,3 +134,5 @@ add the setting manually
 include::upgrade/upgrade-standard.asciidoc[]
 
 include::upgrade/upgrade-migrations.asciidoc[]
+
+include::upgrade/logging-configuration-migration.asciidoc[]

docs/developer/architecture/core/logging-configuration-migration.asciidoc → docs/setup/upgrade/logging-configuration-migration.asciidoc

@@ -1,13 +1,32 @@
 [[logging-configuration-migration]]
 == Logging configuration migration
 
+[float]
+=== Prepare for logging in 8.0
+
+The entire logging system changed to offer a more consistent logging behavior with {es}. The current system differs from the more simplistic, deprecated system and allows you the freedom to customize what, how and where {kib} outputs log messages. While for logging based use cases, we recommend the <<logging-service, logging service guide>>, there are some important considerations:
+
+[float]
+==== Mixing deprecated and current configurations
+{kib} supports mixed usage of the deprecated and current logging configuration styles, however, we recommend changing them all at once to reduce the risk of the systems interfering.
+
+[float]
+==== Rotating log files
+If you use log rotation, be careful when mixing deprecated and current configuration styles as it could lead to unexpected behavior. For example, one system could apply the log file rotation while the other still writes to the original file. Rather switch over to the current system completely.
+
+[float]
+==== Service scripts
+If you are using the service scripts (`kibana.service`), be aware that the deprecated `logging.dest` configuration might be set via CLI options in the `kibana.service` declaration. When `logging.dest` is set in the service scripts, {kib} will continue to write log messages to the "old" file. To switch over to the current logging system, remove the `logging.dest` CLI argument from `/etc/systemd/system/kibana.service` and run `systemctl daemon-reload`.
+
+[float]
+==== Compatibility
 Compatibility with the legacy logging system is assured until the end of the `v7` version. 
-All log messages handled by `root` context are forwarded to the legacy logging service. If you re-write
-root appenders, make sure that it contains `default` appender to provide backward compatibility.
+All log messages handled by `root` context are forwarded to the legacy logging service. If you configure a custom appender for `root`, make sure to include the `default` appender in the appenders array to provide backward compatibility.
 
-NOTE: When you switch to the new logging configuration, you will start seeing duplicate log entries in both formats. 
-These will be removed when the `default` appender is no longer required. If you define an appender for a logger, 
-the log messages aren't handled by the `root` logger anymore and are not forwarded to the legacy logging service.
+[float]
+==== Duplicate log messages
+When you switch to the new logging configuration, you will start seeing duplicate log entries in both legacy and current formats. The legacy format will be removed when the `default` appender is no longer required. 
+To override this behavior for specific log messages, configure an appender for the logger.
 
 [[logging-pattern-format-old-and-new-example]]
 [options="header"]
@@ -54,7 +73,7 @@ the log messages aren't handled by the `root` logger anymore and are not forward
 
 As is the case for any of Kibana's config settings, you can specify your logging configuration via the CLI. For convenience, the `--verbose` and `--silent` flags exist as shortcuts and will continue to be supported beyond v7.
 
-If you wish to override these flags, you can always do so by passing your preferred logging configuration directly to the CLI. For example, with the following configuration:
+If you wish to override these flags, you can always do so by passing your preferred logging configuration directly to the CLI. For example, if {kib} has the following configuration in the `kibana.yml`, you can override the log level with CLI arguments, provided the full logging configuration is passed:
 
 [source,yaml]
 ----
@@ -65,9 +84,12 @@ logging:
       layout:
         type: pattern
         pattern: "[%date][%level] %message"
+    root:
+      appenders: [default, custom]
+      level: warn
 ----
 
-you can override the flags with:
+You can override the flags with:
 
 [options="header"]
 |===

docs/user/reporting/reporting-troubleshooting.asciidoc

@@ -122,7 +122,16 @@ all, the full logs from Reporting will be the first place to look. In `kibana.ym
 
 [source,yaml]
 --------------------------------------------------------------------------------
-logging.root.level: all
+logging:
+  appenders:
+    console_appender:
+      type: console
+      layout:
+        type: pattern
+        highlight: true
+  root:
+    appenders: [default, console_appender]
+    level: debug
 --------------------------------------------------------------------------------
 
 For more information about logging, see <<logging-root-level,Kibana configuration settings>>.