docs: pull beats doc changes into APM (#4476) (#4497)

docs/copied-from-beats/docs/command-reference.asciidoc

@@ -1012,7 +1012,7 @@ default config file, +{beatname_lc}.yml+, is used.
 Enables debugging for the specified selectors. For the selectors, you can
 specify a comma-separated
 list of components, or you can use `-d "*"` to enable debugging for all
-components. For example, `-d "publish"` displays all the "publish" related
+components. For example, `-d "publisher"` displays all the publisher-related
 messages.
 
 *`-e, --e`*::

docs/copied-from-beats/docs/debugging.asciidoc

@@ -27,12 +27,12 @@ platform). You can use a different configuration file by specifying the `-c` fla
 ------------------------------------------------------------
 
 You can increase the verbosity of debug messages by enabling one or more debug
-selectors. For example, to view the published transactions, you can start {beatname_uc}
-with the `publish` selector like this:
+selectors. For example, to view publisher-related messages, start {beatname_uc}
+with the `publisher` selector:
 
 ["source","sh",subs="attributes"]
 ------------------------------------------------------------
-{beatname_lc} -e -d "publish"
+{beatname_lc} -e -d "publisher"
 ------------------------------------------------------------
 
 If you want all the debugging output (fair warning, it's quite a lot), you can

docs/copied-from-beats/docs/howto/load-index-templates.asciidoc

@@ -48,6 +48,10 @@ If the template already exists, it’s not overwritten unless you configure
 [[overwrite-template]]
 === Overwrite an existing index template
 
+WARNING: Do not enable this option for more than one instance of {beatname_uc}. If you start
+multiple instances at the same time, it can overload your {es} with too many
+template update requests.
+
 To overwrite a template that's already loaded into {es}, set:
 
 [source,yaml]

docs/copied-from-beats/docs/loggingconfig.asciidoc

@@ -68,7 +68,7 @@ messages related to event publishing:
 ["source","yaml",subs="attributes"]
 ----
 logging.level: debug
-logging.selectors: ["publish"]
+logging.selectors: ["publisher"]
 ----
 
 The logs generated by {beatname_uc} are written to the CloudWatch log group for
@@ -143,11 +143,22 @@ published. Also logs any warnings, errors, or critical errors.
 ==== `logging.selectors`
 
 The list of debugging-only selector tags used by different {beatname_uc} components.
-Use `*` to enable debug output for all components. For example add `publish` to display
-all the debug messages related to event publishing.
+Use `*` to enable debug output for all components. Use `publisher` to display
+debug messages related to event publishing.
+
+[TIP]
+=====
+The list of available selectors may change between releases, so avoid creating
+tests that depend on specific selectors.
+
+To see which selectors are available, run {beatname_uc} in debug mode
+(set `logging.level: debug` in the configuration). The selector name appears
+after the log level and is enclosed in brackets.
+=====
+
 ifndef::serverless[]
-When starting {beatname_lc}, selectors can be overwritten using the `-d` command
-line option (`-d` also sets the debug log level).
+To override selectors at the command line, use the `-d` global flag (`-d` also
+sets the debug log level). For more information, see <<command-line-options>>.
 endif::serverless[]
 
 [float]

docs/copied-from-beats/docs/monitoring/monitoring-beats.asciidoc

@@ -28,7 +28,6 @@ ifndef::serverless[]
 and sends it directly to your monitoring cluster.
 endif::[]
 
-
 //Commenting out this link temporarily until the general monitoring docs can be
 //updated.
 //To learn about monitoring in general, see

docs/copied-from-beats/docs/repositories.asciidoc

@@ -122,7 +122,7 @@ sudo apt-get update && sudo apt-get install {beatname_pkg}
 --------------------------------------------------
 sudo systemctl enable {beatname_pkg}
 --------------------------------------------------
-
++
 If your system does not use `systemd` then run:
 +
 ["source","sh",subs="attributes"]
@@ -224,7 +224,7 @@ sudo yum install {beatname_pkg}
 --------------------------------------------------
 sudo systemctl enable {beatname_pkg}
 --------------------------------------------------
-
++
 If your system does not use `systemd` then run:
 +
 ["source","sh",subs="attributes"]
@@ -233,4 +233,3 @@ sudo chkconfig --add {beatname_pkg}
 --------------------------------------------------
 
 endif::[]
-

docs/copied-from-beats/docs/security/api-keys.asciidoc

@@ -14,6 +14,8 @@ API key. For different clusters, you need to use an API key per cluster.
 NOTE: For security reasons, we recommend using a unique API key per {beatname_uc} instance.
 You can create as many API keys per user as necessary.
 
+IMPORTANT: Review <<feature-roles>> before creating API keys for {beatname_uc}.
+
 [float]
 [[beats-api-key-publish]]
 === Create an API key for publishing
@@ -41,6 +43,8 @@ POST /_security/api_key
 <1> Name of the API key
 <2> Granted privileges, see <<feature-roles>>
 
+NOTE: See <<privileges-to-publish-events>> for the list of privileges required to publish events.
+
 The return value will look something like this:
 
 [source,console-result,subs="attributes,callouts"]
@@ -89,6 +93,8 @@ POST /_security/api_key
 <1> Name of the API key
 <2> Granted privileges, see <<feature-roles>>
 
+NOTE: See <<privileges-to-publish-monitoring>> for the list of privileges required to send monitoring data.
+
 The return value will look something like this:
 
 [source,console-result,subs="attributes,callouts"]

docs/copied-from-beats/docs/shared-securing-beat.asciidoc

@@ -29,11 +29,13 @@ For secure communication between APM Server and APM Agents, see <<secure-communi
 endif::[]
 
 ifndef::serverless[]
+ifndef::win_only[]
 On Linux, {beatname_uc} can take advantage of secure computing mode to restrict the
 system calls that a process can issue.
 
 * <<linux-seccomp>>
 endif::[]
+endif::[]
 
 // APM HTTPS information
 ifdef::beat-specific-security[]
@@ -70,5 +72,7 @@ endif::[]
 
 // Linux Seccomp
 ifndef::serverless[]
+ifndef::win_only[]
 include::./security/linux-seccomp.asciidoc[]
 endif::[]
+endif::[]

docs/copied-from-beats/docs/shared-ssl-config.asciidoc

@@ -104,7 +104,33 @@ NOTE: SSL settings are disabled if either `enabled` is set to `false` or the
 [float]
 ==== `certificate_authorities`
 
-The list of root certificates for server verifications. If `certificate_authorities` is empty or not set, the trusted certificate authorities of the host system are used.
+The list of root certificates for server verifications. If `certificate_authorities` is empty or not set, the trusted certificate authorities of the host system are used. If `certificate_authorities` is self-signed, the host system needs to trust that CA cert as well.
+By default you can specify a list of file that +{beatname_lc} will read, but you can also embed a certificate directly in the `YAML` configuration:
+
+[source,yaml]
+----
+certificate_authorities:
+  - |
+    -----BEGIN CERTIFICATE-----
+    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
+    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
+    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
+    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
+    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
+    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
+    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
+    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
+    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
+    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
+    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
+    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
+    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
+    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
+    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
+    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
+    sxSmbIUfc2SGJGCJD4I=
+    -----END CERTIFICATE-----
+----
 
 [float]
 [[certificate]]
@@ -117,12 +143,72 @@ require client authentication, the certificate will be loaded, but not requested
 by the server.
 
 When this option is configured, the <<key,`key`>> option is also required.
+The certificate option support embedding of the certificate:
+
+[source,yaml]
+----
+certificate: |
+    -----BEGIN CERTIFICATE-----
+    MIIDCjCCAfKgAwIBAgITJ706Mu2wJlKckpIvkWxEHvEyijANBgkqhkiG9w0BAQsF
+    ADAUMRIwEAYDVQQDDAlsb2NhbGhvc3QwIBcNMTkwNzIyMTkyOTA0WhgPMjExOTA2
+    MjgxOTI5MDRaMBQxEjAQBgNVBAMMCWxvY2FsaG9zdDCCASIwDQYJKoZIhvcNAQEB
+    BQADggEPADCCAQoCggEBANce58Y/JykI58iyOXpxGfw0/gMvF0hUQAcUrSMxEO6n
+    fZRA49b4OV4SwWmA3395uL2eB2NB8y8qdQ9muXUdPBWE4l9rMZ6gmfu90N5B5uEl
+    94NcfBfYOKi1fJQ9i7WKhTjlRkMCgBkWPkUokvBZFRt8RtF7zI77BSEorHGQCk9t
+    /D7BS0GJyfVEhftbWcFEAG3VRcoMhF7kUzYwp+qESoriFRYLeDWv68ZOvG7eoWnP
+    PsvZStEVEimjvK5NSESEQa9xWyJOmlOKXhkdymtcUd/nXnx6UTCFgnkgzSdTWV41
+    CI6B6aJ9svCTI2QuoIq2HxX/ix7OvW1huVmcyHVxyUECAwEAAaNTMFEwHQYDVR0O
+    BBYEFPwN1OceFGm9v6ux8G+DZ3TUDYxqMB8GA1UdIwQYMBaAFPwN1OceFGm9v6ux
+    8G+DZ3TUDYxqMA8GA1UdEwEB/wQFMAMBAf8wDQYJKoZIhvcNAQELBQADggEBAG5D
+    874A4YI7YUwOVsVAdbWtgp1d0zKcPRR+r2OdSbTAV5/gcS3jgBJ3i1BN34JuDVFw
+    3DeJSYT3nxy2Y56lLnxDeF8CUTUtVQx3CuGkRg1ouGAHpO/6OqOhwLLorEmxi7tA
+    H2O8mtT0poX5AnOAhzVy7QW0D/k4WaoLyckM5hUa6RtvgvLxOwA0U+VGurCDoctu
+    8F4QOgTAWyh8EZIwaKCliFRSynDpv3JTUwtfZkxo6K6nce1RhCWFAsMvDZL8Dgc0
+    yvgJ38BRsFOtkRuAGSf6ZUwTO8JJRRIFnpUzXflAnGivK9M13D5GEQMmIl6U9Pvk
+    sxSmbIUfc2SGJGCJD4I=
+    -----END CERTIFICATE-----
+----
+
 
 [float]
 [[key]]
 ==== `key: "/etc/pki/client/cert.key"`
 
 The client certificate key used for client authentication. This option is required if <<certificate,`certificate`>> is specified.
+The key option support embedding of the private key:
+
+[source,yaml]
+----
+key: |
+    -----BEGIN PRIVATE KEY-----
+    MIIEvQIBADANBgkqhkiG9w0BAQEFAASCBKcwggSjAgEAAoIBAQDXHufGPycpCOfI
+    sjl6cRn8NP4DLxdIVEAHFK0jMRDup32UQOPW+DleEsFpgN9/ebi9ngdjQfMvKnUP
+    Zrl1HTwVhOJfazGeoJn7vdDeQebhJfeDXHwX2DiotXyUPYu1ioU45UZDAoAZFj5F
+    KJLwWRUbfEbRe8yO+wUhKKxxkApPbfw+wUtBicn1RIX7W1nBRABt1UXKDIRe5FM2
+    MKfqhEqK4hUWC3g1r+vGTrxu3qFpzz7L2UrRFRIpo7yuTUhEhEGvcVsiTppTil4Z
+    HcprXFHf5158elEwhYJ5IM0nU1leNQiOgemifbLwkyNkLqCKth8V/4sezr1tYblZ
+    nMh1cclBAgMBAAECggEBAKdP5jyOicqknoG9/G564RcDsDyRt64NuO7I6hBg7SZx
+    Jn7UKWDdFuFP/RYtoabn6QOxkVVlydp5Typ3Xu7zmfOyss479Q/HIXxmmbkD0Kp0
+    eRm2KN3y0b6FySsS40KDRjKGQCuGGlNotW3crMw6vOvvsLTlcKgUHF054UVCHoK/
+    Piz7igkDU7NjvJeha53vXL4hIjb10UtJNaGPxIyFLYRZdRPyyBJX7Yt3w8dgz8WM
+    epOPu0dq3bUrY3WQXcxKZo6sQjE1h7kdl4TNji5jaFlvD01Y8LnyG0oThOzf0tve
+    Gaw+kuy17gTGZGMIfGVcdeb+SlioXMAAfOps+mNIwTECgYEA/gTO8W0hgYpOQJzn
+    BpWkic3LAoBXWNpvsQkkC3uba8Fcps7iiEzotXGfwYcb5Ewf5O3Lrz1EwLj7GTW8
+    VNhB3gb7bGOvuwI/6vYk2/dwo84bwW9qRWP5hqPhNZ2AWl8kxmZgHns6WTTxpkRU
+    zrfZ5eUrBDWjRU2R8uppgRImsxMCgYEA2MxuL/C/Ko0d7XsSX1kM4JHJiGpQDvb5
+    GUrlKjP/qVyUysNF92B9xAZZHxxfPWpdfGGBynhw7X6s+YeIoxTzFPZVV9hlkpAA
+    5igma0n8ZpZEqzttjVdpOQZK8o/Oni/Q2S10WGftQOOGw5Is8+LY30XnLvHBJhO7
+    TKMurJ4KCNsCgYAe5TDSVmaj3dGEtFC5EUxQ4nHVnQyCpxa8npL+vor5wSvmsfUF
+    hO0s3GQE4sz2qHecnXuPldEd66HGwC1m2GKygYDk/v7prO1fQ47aHi9aDQB9N3Li
+    e7Vmtdn3bm+lDjtn0h3Qt0YygWj+wwLZnazn9EaWHXv9OuEMfYxVgYKpdwKBgEze
+    Zy8+WDm5IWRjn8cI5wT1DBT/RPWZYgcyxABrwXmGZwdhp3wnzU/kxFLAl5BKF22T
+    kRZ+D+RVZvVutebE9c937BiilJkb0AXLNJwT9pdVLnHcN2LHHHronUhV7vetkop+
+    kGMMLlY0lkLfoGq1AxpfSbIea9KZam6o6VKxEnPDAoGAFDCJm+ZtsJK9nE5GEMav
+    NHy+PwkYsHhbrPl4dgStTNXLenJLIJ+Ke0Pcld4ZPfYdSyu/Tv4rNswZBNpNsW9K
+    0NwJlyMBfayoPNcJKXrH/csJY7hbKviAHr1eYy9/8OL0dHf85FV+9uY5YndLcsDc
+    nygO9KTJuUiBrLr0AHEnqko=
+    -----END PRIVATE KEY-----
+----
 
 [float]
 ==== `key_passphrase`
@@ -143,13 +229,20 @@ The default value is `[TLSv1.1, TLSv1.2, TLSv1.3]`.
 [float]
 ==== `verification_mode`
 
-This option controls whether the client verifies server certificates and host
-names. Valid values are `none` and `full`. If `verification_mode` is set
-to `none`, all server host names and certificates are accepted. In this mode,
-TLS-based connections are susceptible to man-in-the-middle attacks. Use this
-option for testing only.
+Controls the verification of certificates. Valid values are:
+
+ * `full`, which verifies that the provided certificate is signed by a trusted
+authority (CA) and also verifies that the server's hostname (or IP address)
+matches the names identified within the certificate.
+ * `certificate`, which verifies that the provided certificate is signed by a
+trusted authority (CA), but does not perform any hostname verification.
+ * `none`, which performs _no verification_ of the server's certificate. This
+mode disables many of the security benefits of SSL/TLS and should only be used
+after very careful consideration. It is primarily intended as a temporary
+diagnostic mechanism when attempting to resolve TLS errors; its use in
+production environments is strongly discouraged.
 
-The default is `full`.
+The default value is `full`.
 
 [float]
 ==== `cipher_suites`

docs/copied-from-beats/docs/template-config.asciidoc

@@ -23,9 +23,16 @@ endif::[]
 You can adjust the following settings to load your own template or overwrite an
 existing one.
 
-*`setup.template.enabled`*:: Set to false to disable template loading. If set this to false,
+*`setup.template.enabled`*:: Set to false to disable template loading. If this is set to false,
 you must <<load-template-manually,load the template manually>>.
 
+ifndef::apm-server[]
+*`setup.template.type`*:: The type of template to use. Available options: `legacy` (default), index templates
+before Elasticsearch v7.8. Use this to avoid breaking existing deployments. New options are `component`
+and `index`. Selecting `component` loads a component template which can be included in new index templates.
+The option `index` loads the new index template.
+endif::[]
+
 *`setup.template.name`*:: The name of the template. The default is
 +{beatname_lc}+. The {beatname_uc} version is always appended to the given
 name, so the final name is +{beatname_lc}-%{[{beat_version_key}]}+.
@@ -55,7 +62,8 @@ relative path is set, it is considered relative to the config path. See the <<di
 section for details.
 
 *`setup.template.overwrite`*:: A boolean that specifies whether to overwrite the existing template. The default
-is false.
+is false. Do not enable this option if you start more than one instance of {beatname_uc} at the same time. It
+can overload {es} by sending too many template update requests.
 
 *`setup.template.settings`*:: A dictionary of settings to place into the `settings.index` dictionary of the
 Elasticsearch template. For more details about the available Elasticsearch mapping options, please

docs/copied-from-beats/outputs/kafka/docs/kafka.asciidoc

@@ -232,6 +232,19 @@ Set `max_retries` to a value less than 0 to retry until all events are published
 The default is 3.
 endif::[]
 
+===== `backoff.init`
+
+The number of seconds to wait before trying to republish to Kafka
+after a network error. After waiting `backoff.init` seconds, {beatname_uc}
+tries to republish. If the attempt fails, the backoff timer is increased
+exponentially up to `backoff.max`. After a successful publish, the backoff
+timer is reset. The default is 1s.
+
+===== `backoff.max`
+
+The maximum number of seconds to wait before attempting to republish to
+Kafka after a network error. The default is 60s.
+
 ===== `bulk_max_size`
 
 The maximum number of events to bulk in a single Kafka request. The default is 2048.
@@ -261,6 +274,12 @@ The keep-alive period for an active network connection. If 0s, keep-alives are d
 
 Sets the output compression codec. Must be one of `none`, `snappy`, `lz4` and `gzip`. The default is `gzip`.
 
+[IMPORTANT]
+.Known issue with Azure Event Hub for Kafka
+====
+When targeting Azure Event Hub for Kafka, set `compression` to `none` as the provided codecs are not supported.
+====
+
 ===== `compression_level`
 
 Sets the compression level used by gzip. Setting this value to 0 disables compression.
@@ -288,3 +307,11 @@ Configuration options for SSL parameters like the root CA for Kafka connections.
 `-keyalg RSA` argument to ensure it uses a cipher supported by
 https://github.com/Shopify/sarama/wiki/Frequently-Asked-Questions#why-cant-sarama-connect-to-my-kafka-cluster-using-ssl[Filebeat's Kafka library].
 See <<configuration-ssl>> for more information.
+
+===== `kerberos`
+
+beta[]
+
+Configuration options for Kerberos authentication.
+
+See <<configuration-kerberos>> for more information.