From 1c6b89d82f8aa1814747eb8e3b3974419f52f201 Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 27 Apr 2020 19:30:58 +0200 Subject: [PATCH 1/3] Document monitoring workers It doesn't seem to be documented anywhere and means that you suddenly start losing metrics without any obvious reason when you go from monolith to workers (e.g. #7312). --- docs/metrics-howto.md | 25 +++++++++++++++++++++++++ 1 file changed, 25 insertions(+) diff --git a/docs/metrics-howto.md b/docs/metrics-howto.md index 32abb9f44e43..4ab2456e5ba8 100644 --- a/docs/metrics-howto.md +++ b/docs/metrics-howto.md @@ -60,6 +60,31 @@ 1. Restart Prometheus. +## Monitoring workers + +To monitor a Synapse installation using +[workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md), +every worker needs to be monitored independently, in addition to +the main homeserver process. This is because workers don't send +their metrics to the main homeserver process, and therefore +expose their own metrics themselves (if they are configured to +do so). + +To allow collecting metrics from a worker, you need to add a +`metrics` listener to its configuration, by adding the following +under `worker_listeners`: + +```yaml + - type: metrics + bind_address: '' + port: 9101 +``` + +Feel free to change the `bind_address` and `port` parameters +as long as the resulting listener can be reached by prometheus. +With this example, the worker's metrics would then be available +on `http://127.0.0.1:9101`. + ## Renaming of metrics & deprecation of old names in 1.2 Synapse 1.2 updates the Prometheus metrics to match the naming From 883fb051451116623ea41d179380c54ed563d45b Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 27 Apr 2020 19:34:13 +0200 Subject: [PATCH 2/3] Changelog --- changelog.d/7357.doc | 1 + 1 file changed, 1 insertion(+) create mode 100644 changelog.d/7357.doc diff --git a/changelog.d/7357.doc b/changelog.d/7357.doc new file mode 100644 index 000000000000..a3d5616ad263 --- /dev/null +++ b/changelog.d/7357.doc @@ -0,0 +1 @@ +Add documentation on monitoring workers with Prometheus. From 236dbba7e323e38f6bb4264e75970823cc8d7bec Mon Sep 17 00:00:00 2001 From: Brendan Abolivier Date: Mon, 27 Apr 2020 19:46:40 +0200 Subject: [PATCH 3/3] Incorporate review --- docs/metrics-howto.md | 10 +++++----- 1 file changed, 5 insertions(+), 5 deletions(-) diff --git a/docs/metrics-howto.md b/docs/metrics-howto.md index 4ab2456e5ba8..cf69938a2a74 100644 --- a/docs/metrics-howto.md +++ b/docs/metrics-howto.md @@ -66,9 +66,8 @@ To monitor a Synapse installation using [workers](https://github.com/matrix-org/synapse/blob/master/docs/workers.md), every worker needs to be monitored independently, in addition to the main homeserver process. This is because workers don't send -their metrics to the main homeserver process, and therefore -expose their own metrics themselves (if they are configured to -do so). +their metrics to the main homeserver process, but expose them +directly (if they are configured to do so). To allow collecting metrics from a worker, you need to add a `metrics` listener to its configuration, by adding the following @@ -80,8 +79,9 @@ under `worker_listeners`: port: 9101 ``` -Feel free to change the `bind_address` and `port` parameters -as long as the resulting listener can be reached by prometheus. +The `bind_address` and `port` parameters should be set so that +the resulting listener can be reached by prometheus, and they +don't clash with an existing worker. With this example, the worker's metrics would then be available on `http://127.0.0.1:9101`.