Replace HTTP replication with TCP replication (Server side part) #2082

erikjohnston · 2017-03-30T12:42:33Z

This is the server side component of #2069, including the requested changes.

The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one.

This defines the low level TCP replication protocol

The TCP replication protocol streams deltas of who has started or stopped syncing. This is different from the HTTP API which periodically sends the full list of users who are syncing. This commit adds support for the new TCP style of sending deltas.

erikjohnston · 2017-03-30T12:45:31Z

synapse/notifier.py

+        """Returns a deferred which resolves when there is new data for
+        replication to handle.
+        """
+        return self.replication_deferred.observe()


It may be nicer to let the replication resource register a callback rather than bouncing through a deferred?

yes, I think it would. it may be more efficient too.

otherwise I think it needs a make_deferred_yieldable.

erikjohnston · 2017-03-30T12:47:04Z

synapse/replication/tcp/resource.py

+        connections if there are.
+
+        This should get called each time new data is available, even if it
+        is currently being executed, so that nothing gets missed


Maybe it would be a lot clearer if this function looped until there were no changes? Although that is less immediately obvious is correct.

It may be more obvious if we can register this function directly with the notifer, rather than bouncing via a deferred in a loop.

+1 to registering a callback
-1 to looping until there are no changes. It will complicate this function and I don't think it will make anything clearer.

richvdh · 2017-03-30T18:37:53Z

docs/tcp_replication.rst

+The example shows the server accepting a new connection and sending its identity
+with the ``SERVER`` command, followed by the client asking to subscribe to the
+``events`` stream from the token ``53``. The server then periodically sends ``RDATA``
+commands which have the format ``RDATA <stream_name> <token> <row>```, where the


richvdh · 2017-03-30T18:40:14Z

docs/tcp_replication.rst

+Motivation
+----------
+
+The HTTP API used long poll from the workers to the master, this has the problem


This paragraph is going to look out of date real soon. I would go straight for:

Previously the workers used an HTTP long poll mechanism to get updates from the master, which had the problem of causing a lot of duplicate work on the server. This TCP protocol replaces those APIs with the aim of increased efficiency.

[or something]

richvdh · 2017-03-30T18:40:56Z

docs/tcp_replication.rst

+TCP Replication
+===============
+
+This describes the TCP replication protocol that replaces the HTTP protocol.


get rid of this line, it's too vague to be useful

richvdh · 2017-03-30T18:44:17Z

docs/tcp_replication.rst

+
+Blank lines are ignored.
+
+


it would be nice to give a complete list of the commands here, with the command syntax, the direction of transmission, a quick summary and a reference to the section where it is explained in more detail.

richvdh · 2017-03-30T18:45:27Z

docs/tcp_replication.rst

+Reliability
+~~~~~~~~~~~
+
+In general the replication stream should be consisdered an unreliable transport


consisdered

richvdh · 2017-03-30T19:44:40Z

synapse/replication/tcp/resource.py

+        connections if there are.
+
+        This should get called each time new data is available, even if it
+        is currently being executed, so that nothing gets missed


+1 to registering a callback
-1 to looping until there are no changes. It will complicate this function and I don't think it will make anything clearer.

richvdh · 2017-03-30T19:48:52Z

synapse/replication/tcp/__init__.py

+    > RDATA events 55 ["$foo4:bar.com", ...]
+
+The example shows the server accepting a new connection and sending its identity
+with the `SERVER` command, followed by the client asking to subscribe to the


I'm not going to insist you go through and change them, but for future reference: AIUI docstrings are interpreted as RST and really ought to have double-backticks.

richvdh · 2017-03-30T19:54:19Z

synapse/storage/pusher.py

+        """Get all the pushers that have changed between the given tokens.
+
+        Returns:
+            list(tuple): each tuple consists of:


No, it returns a Deferred, which resolves to that lot.

(I'd prefer it if we even put this on @defer.inlineCallbacks, but I won't insist there because it's "obvious". Here though, there is no clue that it actually returns a Deferred, rather than a list.)

richvdh · 2017-03-30T19:55:50Z

synapse/notifier.py

+        """Returns a deferred which resolves when there is new data for
+        replication to handle.
+        """
+        return self.replication_deferred.observe()


otherwise I think it needs a make_deferred_yieldable.

richvdh · 2017-03-30T19:59:30Z

synapse/replication/tcp/streams.py

+        True then limit is provided, otherwise it's not.
+
+        Returns:
+            list(tuple): the first entry in the tuple is the token for that


If it is valid for this function to return a Deferred, say so.

erikjohnston · 2017-03-31T13:30:01Z

I think I've addressed all the issues raised.

I also cheekily added a <last_sync_ms> to the USER_SYNC command (which is the last sync time), as I think a) its more correct and b) is required if we want to batch up the USER_SYNC stuff on the workers to reduce traffic.

This timestamp is used to indicate when the user last sync'd

richvdh

lgtm, modulo the conflict

…pl_tcp_server

@psaavedra

Changes in synapse v0.21.0 (2017-05-18) ======================================= No changes since v0.21.0-rc3 Changes in synapse v0.21.0-rc3 (2017-05-17) =========================================== Features: * Add per user rate-limiting overrides (PR matrix-org#2208) * Add config option to limit maximum number of events requested by ``/sync`` and ``/messages`` (PR matrix-org#2221) Thanks to @psaavedra! Changes: * Various small performance fixes (PR matrix-org#2201, matrix-org#2202, matrix-org#2224, matrix-org#2226, matrix-org#2227, matrix-org#2228, matrix-org#2229) * Update username availability checker API (PR matrix-org#2209, matrix-org#2213) * When purging, don't de-delta state groups we're about to delete (PR matrix-org#2214) * Documentation to check synapse version (PR matrix-org#2215) Thanks to @hamber-dick! * Add an index to event_search to speed up purge history API (PR matrix-org#2218) Bug fixes: * Fix API to allow clients to upload one-time-keys with new sigs (PR matrix-org#2206) Changes in synapse v0.21.0-rc2 (2017-05-08) =========================================== Changes: * Always mark remotes as up if we receive a signed request from them (PR matrix-org#2190) Bug fixes: * Fix bug where users got pushed for rooms they had muted (PR matrix-org#2200) Changes in synapse v0.21.0-rc1 (2017-05-08) =========================================== Features: * Add username availability checker API (PR matrix-org#2183) * Add read marker API (PR matrix-org#2120) Changes: * Enable guest access for the 3pl/3pid APIs (PR matrix-org#1986) * Add setting to support TURN for guests (PR matrix-org#2011) * Various performance improvements (PR matrix-org#2075, matrix-org#2076, matrix-org#2080, matrix-org#2083, matrix-org#2108, matrix-org#2158, matrix-org#2176, matrix-org#2185) * Make synctl a bit more user friendly (PR matrix-org#2078, matrix-org#2127) Thanks @APwhitehat! * Replace HTTP replication with TCP replication (PR matrix-org#2082, matrix-org#2097, matrix-org#2098, matrix-org#2099, matrix-org#2103, matrix-org#2014, matrix-org#2016, matrix-org#2115, matrix-org#2116, matrix-org#2117) * Support authenticated SMTP (PR matrix-org#2102) Thanks @DanielDent! * Add a counter metric for successfully-sent transactions (PR matrix-org#2121) * Propagate errors sensibly from proxied IS requests (PR matrix-org#2147) * Add more granular event send metrics (PR matrix-org#2178) Bug fixes: * Fix nuke-room script to work with current schema (PR matrix-org#1927) Thanks @zuckschwerdt! * Fix db port script to not assume postgres tables are in the public schema (PR matrix-org#2024) Thanks @jerrykan! * Fix getting latest device IP for user with no devices (PR matrix-org#2118) * Fix rejection of invites to unreachable servers (PR matrix-org#2145) * Fix code for reporting old verify keys in synapse (PR matrix-org#2156) * Fix invite state to always include all events (PR matrix-org#2163) * Fix bug where synapse would always fetch state for any missing event (PR matrix-org#2170) * Fix a leak with timed out HTTP connections (PR matrix-org#2180) * Fix bug where we didn't time out HTTP requests to ASes (PR matrix-org#2192) Docs: * Clarify doc for SQLite to PostgreSQL port (PR matrix-org#1961) Thanks @benhylau! * Fix typo in synctl help (PR matrix-org#2107) Thanks @HarHarLinks! * ``web_client_location`` documentation fix (PR matrix-org#2131) Thanks @matthewjwolff! * Update README.rst with FreeBSD changes (PR matrix-org#2132) Thanks @feld! * Clarify setting up metrics (PR matrix-org#2149) Thanks @encks!

erikjohnston added 8 commits March 30, 2017 11:48

Add new storage functions for new replication

24d35ab

The new replication protocol will keep all the streams separate, rather than muxing multiple streams into one.

Add a simple hook to wait for replication traffic

7984708

Make federation send queue take the current position

1188010

Define the various streams we will replicate

8da6f0b

Initial TCP protocol implementation

7450693

This defines the low level TCP replication protocol

Add server side resource for tcp replication

4d7fc7f

Add tcp replication listener type and hook it up

3ba2859

erikjohnston assigned richvdh Mar 30, 2017

erikjohnston commented Mar 30, 2017

View reviewed changes

erikjohnston mentioned this pull request Mar 30, 2017

Replace HTTP replication with TCP replication #2069

Closed

erikjohnston added 2 commits March 30, 2017 13:54

Fix indentation in docs/

31e0fe9

Remove user from process_presence when stops syncing

63fcc42

richvdh suggested changes Mar 30, 2017

View reviewed changes

richvdh assigned erikjohnston and unassigned richvdh Mar 30, 2017

erikjohnston added 3 commits March 31, 2017 11:19

Fix up docs

bfcf016

Add a brief list of commands to docs

b4276a3

Fix up presence

9d0170a

erikjohnston added 2 commits March 31, 2017 15:42

Add a timestamp to USER_SYNC command

36d2b66

This timestamp is used to indicate when the user last sync'd

Use callbacks to notify tcp replication rather than deferreds

1df7c28

erikjohnston force-pushed the erikj/repl_tcp_server branch from 51a80fc to 1df7c28 Compare March 31, 2017 14:43

erikjohnston assigned richvdh and unassigned erikjohnston Apr 3, 2017

Always advance stream tokens

0a6a966

richvdh approved these changes Apr 3, 2017

View reviewed changes

richvdh assigned erikjohnston and unassigned richvdh Apr 3, 2017

Merge branch 'develop' of github.com:matrix-org/synapse into erikj/re…

62b89da

…pl_tcp_server

erikjohnston merged commit 27cc627 into develop Apr 4, 2017

erikjohnston mentioned this pull request Apr 4, 2017

Move to using TCP replication #2097

Merged

erikjohnston deleted the erikj/repl_tcp_server branch October 26, 2017 11:00

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Replace HTTP replication with TCP replication (Server side part) #2082

Replace HTTP replication with TCP replication (Server side part) #2082

erikjohnston commented Mar 30, 2017 •

edited

Loading

erikjohnston Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

erikjohnston Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

erikjohnston Mar 31, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

richvdh Mar 30, 2017

erikjohnston commented Mar 31, 2017 •

edited

Loading

richvdh left a comment

Replace HTTP replication with TCP replication (Server side part) #2082

Replace HTTP replication with TCP replication (Server side part) #2082

Conversation

erikjohnston commented Mar 30, 2017 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Mar 31, 2017 • edited Loading

richvdh left a comment

Choose a reason for hiding this comment

erikjohnston commented Mar 30, 2017 •

edited

Loading

erikjohnston commented Mar 31, 2017 •

edited

Loading