Reduce federation replication traffic #2115

erikjohnston · 2017-04-10T15:47:11Z

This is mainly done by moving the calculation of where to send presence updates from the presence handler to the transaction queue, so we only need to send the presence event (and not the destinations) across the replication connection. Before we were duplicating by sending the full state across once per destination.

erikjohnston · 2017-04-10T15:51:46Z

synapse/federation/transaction_queue.py

+                continue
+
+            host = get_domain_from_id(user_id)
+            hosts_and_states.append(([host], local_states))


The above is copied verbatim from _get_interested_parties

erikjohnston · 2017-04-10T15:52:05Z

synapse/federation/transaction_queue.py

+                    state.user_id: state for state in states
+                })
+
+                preserve_fn(self._attempt_new_transaction)(destination)


The above is copied verbatim from the old send_presence function

erikjohnston · 2017-04-10T15:52:52Z

synapse/federation/transaction_queue.py

@@ -224,17 +227,95 @@ def _send_pdu(self, pdu, destinations):
                self._attempt_new_transaction, destination
            )

-    def send_presence(self, destination, states):
-        if not self.can_send_to(destination):
+    @preserve_fn


Is this an acceptable thing to do style wise?

I think so. I think a comment saying # the caller should not yield on this on this line would help.

richvdh

looks broadly plausible, but I had a bit of a nightmare trying to understand what was going on. A lot of that is because of poor comments in the existing code, so I had to go digging to understand things. Accordingly, I've made a load of requests that you comment things better; I realise that some of them aren't changing, but in the interests of trying to make work in this area less awful in the first place I'd appreciate it if we could take the opportunity to improve things.

I think moving get_interested_parties out would really help.

richvdh · 2017-04-10T17:26:41Z

synapse/federation/transaction_queue.py

@@ -224,17 +227,95 @@ def _send_pdu(self, pdu, destinations):
                self._attempt_new_transaction, destination
            )

-    def send_presence(self, destination, states):
-        if not self.can_send_to(destination):
+    @preserve_fn


I think so. I think a comment saying # the caller should not yield on this on this line would help.

richvdh · 2017-04-10T17:45:08Z

synapse/federation/send_queue.py

@@ -187,18 +190,14 @@ def send_edu(self, destination, edu_type, content, key=None):

        self.notifier.on_new_replication_data()

-    def send_presence(self, destination, states):
+    def send_presence(self, states):
        """As per TransactionQueue"""


can you doc the type of states anyway, please

richvdh · 2017-04-10T17:51:41Z

synapse/handlers/presence.py

        """Sends state updates to remote servers.

        Args:
-            hosts_to_states (dict): Mapping `server_name` -> `[UserPresenceState]`
+            hosts_to_states (list): list(state)


list(UserPresenceState), no?

richvdh · 2017-04-10T18:00:22Z

synapse/app/synchrotron.py

-            states, calculate_remote_hosts=False
-        )
-        room_ids_to_states, users_to_states, _ = parties
+        parties = yield self._get_interested_parties(states)


since we're changing the signature of _get_interested_parties anyway, it would be a good time to rename it to reflect the fact it is being used outside PresenceHandler.

see below: I think _get_interested_parties needs to move anyway.

richvdh · 2017-04-10T18:04:04Z

synapse/handlers/presence.py

-
-            host = get_domain_from_id(user_id)
-            hosts_to_states.setdefault(host, []).extend(local_states)
-
        # TODO: de-dup hosts_to_states, as a single host might have multiple


think this comment is dead

richvdh · 2017-04-11T10:04:08Z

synapse/federation/transaction_queue.py

@@ -78,6 +78,7 @@ def __init__(self, hs):
        self.pending_edus_by_dest = edus = {}

        # Presence needs to be separate as we send single aggragate EDUs
+        self.pending_presence = {}


could you document what all three of these fields mean and their types?

richvdh · 2017-04-11T10:05:27Z

synapse/federation/transaction_queue.py

+    @preserve_fn
+    @defer.inlineCallbacks
+    def send_presence(self, states):
+        """Send the new presence states to the appropriate destinations.


"Start sending" ... "if we are not already sending updates" or something

you didn't like my suggestion?

richvdh · 2017-04-11T10:11:19Z

synapse/federation/transaction_queue.py

+        # hosts in those rooms.
+        room_ids_to_states = {}
+        users_to_states = {}
+        for state in states.itervalues():


please let's not c&p get_interested_parties here. How about moving this lump of code to the store or something?

(For super-mega-bonus points, make it a separate PR which precedes this one...)

richvdh · 2017-04-11T10:12:24Z

synapse/federation/transaction_queue.py

+            for u in plist:
+                users_to_states.setdefault(u, []).append(state)
+
+        hosts_and_states = []


comment on what this is going to be, please.

richvdh · 2017-04-11T10:17:47Z

synapse/federation/transaction_queue.py

+
+        hosts_and_states = []
+        for room_id, states in room_ids_to_states.items():
+            local_states = filter(lambda s: self.is_mine_id(s.user_id), states)


again, why are we filtering, and why would updates for remote users be getting in here? (and could we filter before deriving room_ids_to_states and users_to_states to avoid some work? happy if you want to leave this for now to avoid changing too much at once)

erikjohnston · 2017-04-11T15:10:45Z

I've moved the get_interested_* functions out into standalone functions in the handlers/presence.py file, as it doesn't feel like the sort of thing that belongs in the storage layer

richvdh · 2017-04-12T08:44:59Z

synapse/handlers/presence.py

+    # hosts in those rooms.
+    room_ids_to_states = {}
+    users_to_states = {}
+    for state in states.itervalues():


can we not use get_interested_parties here?

Heh, for some reason I got it stuck in my head that they were different

richvdh · 2017-04-12T08:46:36Z

synapse/federation/transaction_queue.py

+    @preserve_fn
+    @defer.inlineCallbacks
+    def send_presence(self, states):
+        """Send the new presence states to the appropriate destinations.


you didn't like my suggestion?

richvdh · 2017-04-12T08:47:25Z

synapse/federation/transaction_queue.py

+        # to be sent out by user_id. Entries here get processed and put in
+        # pending_presence_by_dest
+        self.pending_presence = {}
+        # Map of destination -> user_id -> UserPresenceState of pending presence


could do with a blank line here

richvdh · 2017-04-12T08:48:15Z

synapse/federation/transaction_queue.py

@@ -78,7 +78,18 @@ def __init__(self, hs):
        self.pending_edus_by_dest = edus = {}

        # Presence needs to be separate as we send single aggragate EDUs


not quite sure what this comment means, any more

richvdh · 2017-04-12T08:50:55Z

synapse/handlers/presence.py

+        each row the list of UserPresenceState should be sent to each
+        destination
+    """
+    hosts_and_states = []  # Final result to return


now that you have an (excellent) description of the return value of the function, this probably doesn't really need a comment, but it's harmless enough

richvdh · 2017-04-12T08:56:48Z

synapse/handlers/presence.py

@@ -669,7 +641,7 @@ def _push_to_remotes(self, states):
        """Sends state updates to remote servers.

        Args:
-            hosts_to_states (list): list(state)
+            hosts_to_states (list(UserPresenceState))


s/hosts_to_states/states/. sorry for not spotting that one before

richvdh · 2017-04-12T09:08:25Z

I've moved the get_interested_* functions out into standalone functions in the handlers/presence.py file, as it doesn't feel like the sort of thing that belongs in the storage layer

Fine by me. Looks much better now.

richvdh

lgtm

@psaavedra

Changes in synapse v0.21.0 (2017-05-18) ======================================= No changes since v0.21.0-rc3 Changes in synapse v0.21.0-rc3 (2017-05-17) =========================================== Features: * Add per user rate-limiting overrides (PR matrix-org#2208) * Add config option to limit maximum number of events requested by ``/sync`` and ``/messages`` (PR matrix-org#2221) Thanks to @psaavedra! Changes: * Various small performance fixes (PR matrix-org#2201, matrix-org#2202, matrix-org#2224, matrix-org#2226, matrix-org#2227, matrix-org#2228, matrix-org#2229) * Update username availability checker API (PR matrix-org#2209, matrix-org#2213) * When purging, don't de-delta state groups we're about to delete (PR matrix-org#2214) * Documentation to check synapse version (PR matrix-org#2215) Thanks to @hamber-dick! * Add an index to event_search to speed up purge history API (PR matrix-org#2218) Bug fixes: * Fix API to allow clients to upload one-time-keys with new sigs (PR matrix-org#2206) Changes in synapse v0.21.0-rc2 (2017-05-08) =========================================== Changes: * Always mark remotes as up if we receive a signed request from them (PR matrix-org#2190) Bug fixes: * Fix bug where users got pushed for rooms they had muted (PR matrix-org#2200) Changes in synapse v0.21.0-rc1 (2017-05-08) =========================================== Features: * Add username availability checker API (PR matrix-org#2183) * Add read marker API (PR matrix-org#2120) Changes: * Enable guest access for the 3pl/3pid APIs (PR matrix-org#1986) * Add setting to support TURN for guests (PR matrix-org#2011) * Various performance improvements (PR matrix-org#2075, matrix-org#2076, matrix-org#2080, matrix-org#2083, matrix-org#2108, matrix-org#2158, matrix-org#2176, matrix-org#2185) * Make synctl a bit more user friendly (PR matrix-org#2078, matrix-org#2127) Thanks @APwhitehat! * Replace HTTP replication with TCP replication (PR matrix-org#2082, matrix-org#2097, matrix-org#2098, matrix-org#2099, matrix-org#2103, matrix-org#2014, matrix-org#2016, matrix-org#2115, matrix-org#2116, matrix-org#2117) * Support authenticated SMTP (PR matrix-org#2102) Thanks @DanielDent! * Add a counter metric for successfully-sent transactions (PR matrix-org#2121) * Propagate errors sensibly from proxied IS requests (PR matrix-org#2147) * Add more granular event send metrics (PR matrix-org#2178) Bug fixes: * Fix nuke-room script to work with current schema (PR matrix-org#1927) Thanks @zuckschwerdt! * Fix db port script to not assume postgres tables are in the public schema (PR matrix-org#2024) Thanks @jerrykan! * Fix getting latest device IP for user with no devices (PR matrix-org#2118) * Fix rejection of invites to unreachable servers (PR matrix-org#2145) * Fix code for reporting old verify keys in synapse (PR matrix-org#2156) * Fix invite state to always include all events (PR matrix-org#2163) * Fix bug where synapse would always fetch state for any missing event (PR matrix-org#2170) * Fix a leak with timed out HTTP connections (PR matrix-org#2180) * Fix bug where we didn't time out HTTP requests to ASes (PR matrix-org#2192) Docs: * Clarify doc for SQLite to PostgreSQL port (PR matrix-org#1961) Thanks @benhylau! * Fix typo in synctl help (PR matrix-org#2107) Thanks @HarHarLinks! * ``web_client_location`` documentation fix (PR matrix-org#2131) Thanks @matthewjwolff! * Update README.rst with FreeBSD changes (PR matrix-org#2132) Thanks @feld! * Clarify setting up metrics (PR matrix-org#2149) Thanks @encks!

erikjohnston force-pushed the erikj/dedupe_federation_repl branch from dffee64 to 29574fd Compare April 10, 2017 15:49

erikjohnston commented Apr 10, 2017

View reviewed changes

erikjohnston requested a review from richvdh April 10, 2017 15:56

erikjohnston assigned richvdh Apr 10, 2017

richvdh suggested changes Apr 11, 2017

View reviewed changes

richvdh assigned erikjohnston and unassigned richvdh Apr 11, 2017

erikjohnston added 4 commits April 11, 2017 15:15

Comments

b9b72bc

Move get_interested_remotes back to presence handler

6308ac4

Comments

2be8a28

Move get_interested_parties

414522a

erikjohnston force-pushed the erikj/dedupe_federation_repl branch from 39d527d to 414522a Compare April 11, 2017 14:33

erikjohnston added 2 commits April 11, 2017 15:35

Comment

a8c8e4e

Move get_presence_list_* to SlaveStore

9c712a3

erikjohnston assigned richvdh and unassigned erikjohnston Apr 11, 2017

richvdh suggested changes Apr 12, 2017

View reviewed changes

richvdh assigned erikjohnston and unassigned richvdh Apr 12, 2017

erikjohnston added 2 commits April 12, 2017 10:16

Reuse get_interested_parties

c7ddb5e

Comment

1745069

erikjohnston assigned richvdh and unassigned erikjohnston Apr 12, 2017

richvdh approved these changes Apr 12, 2017

View reviewed changes

richvdh removed their assignment Apr 12, 2017

richvdh assigned erikjohnston Apr 12, 2017

erikjohnston merged commit 247c736 into develop Apr 12, 2017

erikjohnston deleted the erikj/dedupe_federation_repl branch October 26, 2017 11:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Reduce federation replication traffic #2115

Reduce federation replication traffic #2115

erikjohnston commented Apr 10, 2017

erikjohnston Apr 10, 2017

erikjohnston Apr 10, 2017

erikjohnston Apr 10, 2017

richvdh Apr 10, 2017

richvdh left a comment

richvdh Apr 10, 2017

richvdh Apr 10, 2017

richvdh Apr 10, 2017

richvdh Apr 10, 2017

richvdh Apr 10, 2017

richvdh Apr 11, 2017

richvdh Apr 11, 2017

richvdh Apr 12, 2017

richvdh Apr 11, 2017

richvdh Apr 11, 2017

richvdh Apr 11, 2017

erikjohnston commented Apr 11, 2017

richvdh Apr 12, 2017

erikjohnston Apr 12, 2017

richvdh Apr 12, 2017

richvdh Apr 12, 2017

richvdh Apr 12, 2017

richvdh Apr 12, 2017

richvdh Apr 12, 2017

richvdh commented Apr 12, 2017

richvdh left a comment

		@@ -78,7 +78,18 @@ def __init__(self, hs):
		self.pending_edus_by_dest = edus = {}

		# Presence needs to be separate as we send single aggragate EDUs

Reduce federation replication traffic #2115

Reduce federation replication traffic #2115

Conversation

erikjohnston commented Apr 10, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Apr 11, 2017

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh commented Apr 12, 2017

richvdh left a comment

Choose a reason for hiding this comment