From 6e0c1e36317d3f21673d024312c2528da3ede9d8 Mon Sep 17 00:00:00 2001 From: Andrew Cholakian Date: Tue, 4 Sep 2018 22:04:05 -0500 Subject: [PATCH 1/3] Document heartbeat config reloading --- heartbeat/docs/heartbeat-options.asciidoc | 170 +++++++++++++--------- 1 file changed, 104 insertions(+), 66 deletions(-) diff --git a/heartbeat/docs/heartbeat-options.asciidoc b/heartbeat/docs/heartbeat-options.asciidoc index b6aa9bfa7bac..faddb91a176f 100644 --- a/heartbeat/docs/heartbeat-options.asciidoc +++ b/heartbeat/docs/heartbeat-options.asciidoc @@ -5,17 +5,22 @@ Set up Heartbeat monitors ++++ -The `heartbeat.monitors` section of the +heartbeat.yml+ config file specifies -the list of `monitors` that Heartbeat uses to check your remote hosts to -determine if they are available. Each `monitor` item begins with a dash (-) and -specifies the type of monitor to use, the hosts to check, and other settings -that control Heartbeat behavior. +Heartbeat is configured by defining a set of `monitors` to check your remote hosts. +These can be specified either inside the +heartbeat.yml+ config file, or in external +files. One advantage of using external files is that these can be automatically +reloaded without stopping the Heartbeat process. -The following example configures three monitors, `icmp`, `tcp`, and `http`, and -demonstrates how to use TCP Echo and HTTP response verification: +Each `monitor` item is an entry in a yaml list, and so begins with a dash (-). +You can define the type of monitor to use, the hosts to check, and other +optional settings that control Heartbeat behavior. + +The following example configures three monitors checking via the `icmp`, `tcp`, and `http` +protocols directly inside the +heartbeat.yml+ file, and demonstrates how to use TCP Echo +and HTTP response verification: [source,yaml] ---------------------------------------------------------------------- +# heartbeat.yml heartbeat.monitors: - type: icmp schedule: '*/5 * * * * * *' @@ -33,14 +38,44 @@ heartbeat.scheduler: limit: 10 ---------------------------------------------------------------------- +Using the +heartbeat.yml+ configuration file is convenient, but has two drawbacks: +it can become hard to manage with large numbers of monitors, and it will not reload +heartbeat automatically when its contents changes. + +Define monitors via the +heartbeat.config.monitors+ to prevent those issues from +happening to you. To do so you would instead have your +heartbeat.yml+ file contain the following: + +[source,yaml] +---------------------------------------------------------------------- +# heartbeat.yml +heartbeat.config.monitors: + # Directory + glob pattern to search for configuration files + path: /path/to/my/monitors.d/*.yml + # How often to check for changes + reload.period: 1s +---------------------------------------------------------------------- + +Then, define one or more files in the directory pointed to by `heartbeat.config.monitors.path`. +You may specify multiple monitors in a given file if you like. The contents of these files is +monitor definitions only, e.g. what is normally under the `heartbeat.monitors` section of ++heartbeat.yml+. See below for an example + +[source,yaml] +---------------------------------------------------------------------- +# /path/to/my/monitors.d/localhost_service_check.yml +- type: http + schedule: '@every 5s' + urls: ["http://localhost:80/service/status"] + check.response.status: 200 +---------------------------------------------------------------------- + [float] [[monitor-options]] === Monitor options -You can specify the following options in the `heartbeat.monitors` section of the -+heartbeat.yml+ config file. These options are the same for all monitors. -Each monitor type has additional configuration options that are specific to that -monitor type. +You can specify the following options when defining a Heartbeat monitor in any location. +These options are the same for all monitors. Each monitor type has additional configuration +options that are specific to that monitor type. [float] [[monitor-type]] @@ -160,61 +195,6 @@ A list of processors to apply to the data generated by the monitor. See <> for information about specifying processors in your config. -[float] -[[monitor-watch-poll-file]] -==== `watch.poll_file` - -experimental[] - -The JSON file to watch for additional monitor configurations. The JSON file can -contain multiple objects, each of which specifies a different monitor config. -Heartbeat checks this file periodically and starts a new monitor instance for -each new JSON object added to the file. For example, imagine that you add -10 new entries to the JSON file, each for a different hostname. When Heartbeat -picks up the changes in the file, it merges the original config -(`heartbeat.yml`) plus the JSON objects, and starts a monitor for each new host -that you've configured. If you delete an object from the JSON file and it -doesn't exist in the main config, Heartbeat stops the monitor instance running -for that object. - -Each monitor has a unique ID that's based on parameters like protocol, host, -and port. If two monitors have the same ID, Heartbeat uses the settings that -are defined in the last JSON object of the merged config. This means that -you can specify settings in the JSON file that overwrite the settings in -the main config. In this way, the configuration that you specify for the -monitor in the main Heartbeat config file acts like a default config that you -can live-reconfigure by specifying additional configurations in the external -JSON file. - -Example configuration: - -[source, yaml] -------------------------------------------------------------------------------- -heartbeat.monitors: -- type: tcp - schedule: '*/5 * * * * * *' - hosts: ["myhost"] - watch.poll_file: - path: {path.config}/monitors/dynamic.json - interval: 5s -------------------------------------------------------------------------------- - -*`path`*:: Specifies the path to the JSON file to check for updates. -*`interval`*:: Specifies how often Heartbeat checks the file for changes. - -To reconfigure the settings specified in the example config, you could define -the following JSON objects in `dynamic.json`: - -[source, json] -------------------------------------------------------------------------------- -{"hosts": ["myhost:1234"], "schedule": "*/15 * * * * * *"} <1> -{"hosts": ["tls://otherhost:479"], "ssl.certificate_authorities": ["path/to/ca/file.pem"]} <2> -------------------------------------------------------------------------------- -<1> Upon detecting the changes, Heartbeat stops the old monitor and then -restarts it with a schedule of 15 seconds between checks. -<2> Heartbeat starts a new monitor that uses a TLS-based connection with a -custom CA certificate. - [float] [[monitor-icmp-options]] === ICMP options @@ -556,5 +536,63 @@ specify for `limit` should be below the configured ulimit. The timezone for the scheduler. By default the scheduler uses localtime. +[float] +[[monitor-watch-poll-file]] +==== `watch.poll_file` + +experimental[] + +THIS EXPERIMENTAL FEATURE IS NOW DEPRECATED. IT WILL BE REMOVED IN A FUTURE +RELEASE. It is replaced by using dynamic reloading via the +`heartbeat.config.monitors` option. + +The JSON file to watch for additional monitor configurations. The JSON file can +contain multiple objects, each of which specifies a different monitor config. +Heartbeat checks this file periodically and starts a new monitor instance for +each new JSON object added to the file. For example, imagine that you add +10 new entries to the JSON file, each for a different hostname. When Heartbeat +picks up the changes in the file, it merges the original config +(`heartbeat.yml`) plus the JSON objects, and starts a monitor for each new host +that you've configured. If you delete an object from the JSON file and it +doesn't exist in the main config, Heartbeat stops the monitor instance running +for that object. + +Each monitor has a unique ID that's based on parameters like protocol, host, +and port. If two monitors have the same ID, Heartbeat uses the settings that +are defined in the last JSON object of the merged config. This means that +you can specify settings in the JSON file that overwrite the settings in +the main config. In this way, the configuration that you specify for the +monitor in the main Heartbeat config file acts like a default config that you +can live-reconfigure by specifying additional configurations in the external +JSON file. + +Example configuration: + +[source, yaml] +------------------------------------------------------------------------------- +heartbeat.monitors: +- type: tcp + schedule: '*/5 * * * * * *' + hosts: ["myhost"] + watch.poll_file: + path: {path.config}/monitors/dynamic.json + interval: 5s +------------------------------------------------------------------------------- + +*`path`*:: Specifies the path to the JSON file to check for updates. +*`interval`*:: Specifies how often Heartbeat checks the file for changes. + +To reconfigure the settings specified in the example config, you could define +the following JSON objects in `dynamic.json`: + +[source, json] +------------------------------------------------------------------------------- +{"hosts": ["myhost:1234"], "schedule": "*/15 * * * * * *"} <1> +{"hosts": ["tls://otherhost:479"], "ssl.certificate_authorities": ["path/to/ca/file.pem"]} <2> +------------------------------------------------------------------------------- +<1> Upon detecting the changes, Heartbeat stops the old monitor and then +restarts it with a schedule of 15 seconds between checks. +<2> Heartbeat starts a new monitor that uses a TLS-based connection with a +custom CA certificate. From f1a1d3f70af46a52df81af9a2e8da51813e8baed Mon Sep 17 00:00:00 2001 From: Andrew Cholakian Date: Wed, 19 Sep 2018 11:23:24 -0500 Subject: [PATCH 2/3] Incorporate @dede's feedback --- heartbeat/docs/heartbeat-options.asciidoc | 20 +++++++++----------- 1 file changed, 9 insertions(+), 11 deletions(-) diff --git a/heartbeat/docs/heartbeat-options.asciidoc b/heartbeat/docs/heartbeat-options.asciidoc index faddb91a176f..1b8ab1ba350b 100644 --- a/heartbeat/docs/heartbeat-options.asciidoc +++ b/heartbeat/docs/heartbeat-options.asciidoc @@ -2,13 +2,14 @@ == Set up monitors ++++ -Set up Heartbeat monitors +Set up {beatname_uc} monitors ++++ -Heartbeat is configured by defining a set of `monitors` to check your remote hosts. -These can be specified either inside the +heartbeat.yml+ config file, or in external -files. One advantage of using external files is that these can be automatically -reloaded without stopping the Heartbeat process. +To configure {beatname_uc} define a set of `monitors` to check your remote hosts. +Specify monitors either directly inside the +heartbeat.yml+ config file, or in external +dynamically loaded files located in the directory referenced by `heartbeat.config.monitors.path`. +One advantage of using external files is that these can be automatically reloaded +without stopping the Heartbeat process. Each `monitor` item is an entry in a yaml list, and so begins with a dash (-). You can define the type of monitor to use, the hosts to check, and other @@ -540,15 +541,12 @@ The timezone for the scheduler. By default the scheduler uses localtime. [[monitor-watch-poll-file]] ==== `watch.poll_file` -experimental[] - -THIS EXPERIMENTAL FEATURE IS NOW DEPRECATED. IT WILL BE REMOVED IN A FUTURE -RELEASE. It is replaced by using dynamic reloading via the -`heartbeat.config.monitors` option. +deprecated[6.5.0,Replaced by using dynamic reloading via the +`heartbeat.config.monitors` option.] The JSON file to watch for additional monitor configurations. The JSON file can contain multiple objects, each of which specifies a different monitor config. -Heartbeat checks this file periodically and starts a new monitor instance for +{beatname_uc} checks this file periodically and starts a new monitor instance for each new JSON object added to the file. For example, imagine that you add 10 new entries to the JSON file, each for a different hostname. When Heartbeat picks up the changes in the file, it merges the original config From 112c437319e62349ad308acf04ae171fc4b5d432 Mon Sep 17 00:00:00 2001 From: Andrew Cholakian Date: Wed, 19 Sep 2018 11:24:31 -0500 Subject: [PATCH 3/3] Replace references to Heartbeat with {beatname_uc --- heartbeat/docs/heartbeat-options.asciidoc | 46 +++++++++++------------ 1 file changed, 23 insertions(+), 23 deletions(-) diff --git a/heartbeat/docs/heartbeat-options.asciidoc b/heartbeat/docs/heartbeat-options.asciidoc index 1b8ab1ba350b..6587cf1308a0 100644 --- a/heartbeat/docs/heartbeat-options.asciidoc +++ b/heartbeat/docs/heartbeat-options.asciidoc @@ -9,11 +9,11 @@ To configure {beatname_uc} define a set of `monitors` to check your remote hosts Specify monitors either directly inside the +heartbeat.yml+ config file, or in external dynamically loaded files located in the directory referenced by `heartbeat.config.monitors.path`. One advantage of using external files is that these can be automatically reloaded -without stopping the Heartbeat process. +without stopping the {beatname_uc} process. Each `monitor` item is an entry in a yaml list, and so begins with a dash (-). You can define the type of monitor to use, the hosts to check, and other -optional settings that control Heartbeat behavior. +optional settings that control {beatname_uc} behavior. The following example configures three monitors checking via the `icmp`, `tcp`, and `http` protocols directly inside the +heartbeat.yml+ file, and demonstrates how to use TCP Echo @@ -74,7 +74,7 @@ monitor definitions only, e.g. what is normally under the `heartbeat.monitors` s [[monitor-options]] === Monitor options -You can specify the following options when defining a Heartbeat monitor in any location. +You can specify the following options when defining a {beatname_uc} monitor in any location. These options are the same for all monitors. Each monitor type has additional configuration options that are specific to that monitor type. @@ -118,7 +118,7 @@ A cron-like expression that specifies the task schedule. For example: * `*/5 * * * * * *` runs the task every 5 seconds (for example, at 10:00:00, 10:00:05, and so on). -* `@every 5s` runs the task every 5 seconds from the time when Heartbeat was +* `@every 5s` runs the task every 5 seconds from the time when {beatname_uc} was started. The `schedule` option uses a cron-like syntax based on https://github.com/gorhill/cronexpr#implementation[this `cronexpr` implementation], @@ -154,7 +154,7 @@ ping every IP address for the specified hostname. The default is `any`. The total running time for each ping test. This is the total time allowed for testing the connection and exchanging data. The default is 16 seconds (16s). -If the timeout is exceeded, Heartbeat publishes a `service-down` event. If the +If the timeout is exceeded, {beatname_uc} publishes a `service-down` event. If the value specified for `timeout` is greater than `schedule`, intermediate checks will not be executed by the scheduler. @@ -178,7 +178,7 @@ will be overwritten by the value declared here. If this option is set to true, the custom <> are stored as top-level fields in the output document instead of being grouped under a `fields` sub-dictionary. If the custom field names conflict with other -field names added by Heartbeat, then the custom fields overwrite the other +field names added by {beatname_uc}, then the custom fields overwrite the other fields. [float] @@ -200,7 +200,7 @@ processors in your config. [[monitor-icmp-options]] === ICMP options -These options configure Heartbeat to use ICMP (v4 and v6) Echo Requests to check +These options configure {beatname_uc} to use ICMP (v4 and v6) Echo Requests to check the configured hosts. These options are valid when the <> is `icmp`. @@ -221,7 +221,7 @@ second (1s). [[monitor-tcp-options]] === TCP options -These options configure Heartbeat to connect via TCP and optionally verify the +These options configure {beatname_uc} to connect via TCP and optionally verify the endpoint by sending and/or receiving a custom payload. These options are valid when the <> is `tcp`. @@ -233,16 +233,16 @@ A list of hosts to ping. The entries in the list can be: * A plain host name, such as `localhost`, or an IP address. If you specify this option, you must also specify a value for <>. If the -monitor is <>, Heartbeat establishes an +monitor is <>, {beatname_uc} establishes an SSL/TLS-based connection. Otherwise, it establishes a plain TCP connection. -* A hostname and port, such as `localhost:12345`. Heartbeat connects +* A hostname and port, such as `localhost:12345`. {beatname_uc} connects to the port on the specified host. If the monitor is -<>, Heartbeat establishes an +<>, {beatname_uc} establishes an SSL/TLS-based connection. Otherwise, it establishes a TCP connection. * A full URL using the syntax `scheme://:[port]`, where: ** `scheme` is one of `tcp`, `plain`, `ssl` or `tls`. If `tcp` or `plain` is -specified, Heartbeat establishes a TCP connection even if the monitor is -configured to use SSL. If `tls` or `ssl` is specified, Heartbeat establishes +specified, {beatname_uc} establishes a TCP connection even if the monitor is +configured to use SSL. If `tls` or `ssl` is specified, {beatname_uc} establishes an SSL connection. However, if the monitor is not configured to use SSL, the system defaults are used (currently not supported on Windows). ** `host` is the hostname. @@ -347,7 +347,7 @@ Also see <> for a full description of the `ssl` options. [[monitor-http-options]] === HTTP options -These options configure Heartbeat to connect via HTTP and optionally verify that +These options configure {beatname_uc} to connect via HTTP and optionally verify that the host returns the expected response. These options are valid when the <> is `http`. @@ -523,10 +523,10 @@ address via DNS. [[heartbeat-scheduler-limit]] ==== `limit` -The number of concurrent I/O tasks that Heartbeat is allowed to execute. If set +The number of concurrent I/O tasks that {beatname_uc} is allowed to execute. If set to 0, there is no limit. The default is 0. -Most operating systems set a file descriptor limit of 1024. For Heartbeat to +Most operating systems set a file descriptor limit of 1024. For {beatname_uc} to operate correctly and not accidentally block libbeat output, the value that you specify for `limit` should be below the configured ulimit. @@ -548,19 +548,19 @@ The JSON file to watch for additional monitor configurations. The JSON file can contain multiple objects, each of which specifies a different monitor config. {beatname_uc} checks this file periodically and starts a new monitor instance for each new JSON object added to the file. For example, imagine that you add -10 new entries to the JSON file, each for a different hostname. When Heartbeat +10 new entries to the JSON file, each for a different hostname. When {beatname_uc} picks up the changes in the file, it merges the original config (`heartbeat.yml`) plus the JSON objects, and starts a monitor for each new host that you've configured. If you delete an object from the JSON file and it -doesn't exist in the main config, Heartbeat stops the monitor instance running +doesn't exist in the main config, {beatname_uc} stops the monitor instance running for that object. Each monitor has a unique ID that's based on parameters like protocol, host, -and port. If two monitors have the same ID, Heartbeat uses the settings that +and port. If two monitors have the same ID, {beatname_uc} uses the settings that are defined in the last JSON object of the merged config. This means that you can specify settings in the JSON file that overwrite the settings in the main config. In this way, the configuration that you specify for the -monitor in the main Heartbeat config file acts like a default config that you +monitor in the main {beatname_uc} config file acts like a default config that you can live-reconfigure by specifying additional configurations in the external JSON file. @@ -578,7 +578,7 @@ heartbeat.monitors: ------------------------------------------------------------------------------- *`path`*:: Specifies the path to the JSON file to check for updates. -*`interval`*:: Specifies how often Heartbeat checks the file for changes. +*`interval`*:: Specifies how often {beatname_uc} checks the file for changes. To reconfigure the settings specified in the example config, you could define the following JSON objects in `dynamic.json`: @@ -588,9 +588,9 @@ the following JSON objects in `dynamic.json`: {"hosts": ["myhost:1234"], "schedule": "*/15 * * * * * *"} <1> {"hosts": ["tls://otherhost:479"], "ssl.certificate_authorities": ["path/to/ca/file.pem"]} <2> ------------------------------------------------------------------------------- -<1> Upon detecting the changes, Heartbeat stops the old monitor and then +<1> Upon detecting the changes, {beatname_uc} stops the old monitor and then restarts it with a schedule of 15 seconds between checks. -<2> Heartbeat starts a new monitor that uses a TLS-based connection with a +<2> {beatname_uc} starts a new monitor that uses a TLS-based connection with a custom CA certificate.