Metrics (esp. synapse_notifier_users) and workers

[eMPee584]

Description

Is the assumption correct that synapse_notifier_users should reflect the number of active user devices at a given time?
Since activating workers, synapse_notifier_users has turned to 0.0, leaving us without this vital measure of server activity.. With 7k+ users atm... 😅 💦
Here's a dump of our metrics.gz..

Steps to reproduce

• enable workers
• check http://localhost:9100/_synapse/metrics
• see synapse_notifier_users 0.0
• wonder

Version information

• Homeserver: https://matrix.tu-dresden.de
• Version: python_version: 3.7.7, server_version: 1.12.3
• Install method: https://github.com/spantaleev/matrix-docker-ansible-deploy
• Platform: debian buster VM

[babolivier]

Hey there, which worker are you trying to retrieve this metric from? From what I can see, only the synchrotron (synapse.app.synchrotron) seems to be updating it, so any other worker (or the master process) will be returning 0.0.

To retrieve metrics from a worker, just add the following to its configuration file, under the worker_listeners section:

 - type: metrics
   bind_address: ''
   port: 9101

(feel free to edit the bind_address and port parameters however you want as long as the resulting listener is accessible from prometheus/beamium/etc.)

[eMPee584]

Ok thanks for the pointer.. we were receiving the URL from the main homeserver process..
so different workers provide different metrics fragment? 🤔
And what about multiple synchrotrons, will they each have their own stats?

[babolivier]

Gah, sorry, I missed your reply.

so different workers provide different metrics fragment?

Yes, different workers will handle different data and thus they'll all expose different timeseries.

And what about multiple synchrotrons, will they each have their own stats?

Yup.

The general rule here is that if you want to monitor a cluster of workers you should monitor every worker, not only the master. This isn't clear in the documentation so I've opened #7357 to address that.

I'm going to close this issue now. If you have further comments or questions, feel free to join #synapse:matrix.org and share them there, as it is our main support channel :)