This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Running redis mode (on develop) causes high CPU usage #7334
Labels
A-Workers
Problems related to running Synapse in Worker Mode (or replication)
Comments
I suspect its due to the fact that it seems to be sending out about 1kHz of |
Switching off Redis (but running the same commit) seemed to reduce CPU, and stop spamming |
I believe the cause of this is if the fact in current TCP mode if a worker detects a remote server has come back online it sends a |
erikjohnston
added a commit
that referenced
this issue
Apr 27, 2020
For direct TCP connections we need the master to relay REMOTE_SERVER_UP commands to the other connections so that all instances get notified about it. The old implementation just relayed to all connections, assuming that sending back to the original sender of the command was safe. This is not true for redis, where commands sent get echoed back to the sender, which was causing master to effectively infinite loop sending and then re-receiving REMOTE_SERVER_UP commands that it sent. The fix is to ensure that we only relay to *other* connections and not to the connection we received the notification from. Fixes #7334.
I can confirm #7352 fixes the issue for me. |
erikjohnston
added a commit
that referenced
this issue
Apr 29, 2020
For direct TCP connections we need the master to relay REMOTE_SERVER_UP commands to the other connections so that all instances get notified about it. The old implementation just relayed to all connections, assuming that sending back to the original sender of the command was safe. This is not true for redis, where commands sent get echoed back to the sender, which was causing master to effectively infinite loop sending and then re-receiving REMOTE_SERVER_UP commands that it sent. The fix is to ensure that we only relay to *other* connections and not to the connection we received the notification from. Fixes #7334.
phil-flex
pushed a commit
to phil-flex/synapse
that referenced
this issue
Jun 16, 2020
For direct TCP connections we need the master to relay REMOTE_SERVER_UP commands to the other connections so that all instances get notified about it. The old implementation just relayed to all connections, assuming that sending back to the original sender of the command was safe. This is not true for redis, where commands sent get echoed back to the sender, which was causing master to effectively infinite loop sending and then re-receiving REMOTE_SERVER_UP commands that it sent. The fix is to ensure that we only relay to *other* connections and not to the connection we received the notification from. Fixes matrix-org#7334.
richvdh
added
the
A-Workers
Problems related to running Synapse in Worker Mode (or replication)
label
Feb 16, 2022
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
The CPU graph for my set of workers:
The GC graph:
Something has changed between 1.12.3 and 2e3b9a0 that has caused this to skyrocket.
Otherwise, the homeserver seems to work in terms of federating and sending messages. It's just obsessed with GCing right now.
Memory usage has remained a bit low:
The text was updated successfully, but these errors were encountered: