Precompute joined hosts and store in Redis #9198

erikjohnston · 2021-01-21T19:29:52Z

This is a fairly noddy implementation, but hopefully makes sense as a first cut

synapse/federation/sender/__init__.py

clokep · 2021-01-22T12:20:14Z

synapse/handlers/message.py

@@ -432,6 +432,9 @@ def __init__(self, hs: "HomeServer"):

        self._ephemeral_events_enabled = hs.config.enable_ephemeral_messages

+        self._external_cache = hs.get_external_cache()
+        self._use_external_cache = hs.config.redis.redis_enabled


I wonder if this should be redis enabled and more than one federation sender?

Or at the very least if the federation sending is done by the main process it probably doesn't make sense to use this?

Maybe, though it is worth noting that it's a) probably cheaper to calculate it on event creator, as you'll already have the state in the cache, and b) means that you can run multiple event creators to split up the work without having to run multiple federation senders.

Ah, right that makes sense. I guess it is a reasonable assumption that anyone running workers would be running at least one of an event creator or federation sender?

Yeah, and its really not the end of the world if they aren't.

clokep · 2021-01-22T12:24:46Z

synapse/replication/tcp/external_cache.py

+    def _get_redis_key(self, cache_name: str, key: str) -> str:
+        return "cache_v1:%s:%s" % (cache_name, key)
+
+    async def set_cache(


Our other caches seem to name these methods as simply set and get

I also wonder if instead of expire_seconds we should do expiry_ms as the ExpiringCache does.

Annoyingly the ExpiringCache uses expiry_ms (with a y). I don't know if matters much.

Also do you have thoughts on set/get vs. set_cache/get_cache?

I KNEW THERE WAS SOMETHING I FORGOT TO DO

synapse/replication/tcp/external_cache.py

synapse/federation/sender/__init__.py

clokep · 2021-01-22T17:24:44Z

Do we need any tests for this or would they just be mocking out the world?

clokep · 2021-01-22T18:51:58Z

Do we need any tests for this or would they just be mocking out the world?

We talked about this in #synapse-dev:matrix.org and decided that we wouldn't really be testing anything with unit tests. This is testing via worker mode of sytest though! 👍

clokep

Per the changes to get / set for the API.

synapse/replication/tcp/external_cache.py

richvdh · 2021-01-25T15:56:56Z

synapse/replication/tcp/external_cache.py

+        if not result:
+            return None
+
+        # For some reason the integers get magically converted back to integers


at a quick glance, txredisapi encodes integers as strings (starting with $<len>\r\n) rather than ints (starting with :) for the redis protocol, so the json-encoding of an int is the same as the int itself.

possibly to counter that bit of hackery, txredisapi also has a convertNumbers setting, which is enabled for us, which will make it try to convert things-that-look-like-numbers back into numbers. That sounds like a CPU sink and a thing we should turn off...

Oh wow, that's quite special. Yeah, I agree we should turn it off.

I'll do it in a separate PR though I think

…_hosts_redis

This ensures that when we call `resolve_state_groups` again we get the assigned state group returned.

erikjohnston · 2021-01-26T12:00:00Z

I've added a few more commits with the aim of getting a better cache hit ratio, in a SyTest run it looks like we're now at 100% cache hit ratio.

clokep

LGTM!

synapse/handlers/message.py

erikjohnston · 2021-01-26T13:57:39Z

Thanks @clokep !

@Bubu

Synapse 1.28.0 (2021-02-25) =========================== Note that this release drops support for ARMv7 in the official Docker images, due to repeated problems building for ARMv7 (and the associated maintenance burden this entails). This release also fixes the documentation included in v1.27.0 around the callback URI for SAML2 identity providers. If your server is configured to use single sign-on via a SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. Internal Changes ---------------- - Revert change in v1.28.0rc1 to remove the deprecated SAML endpoint. ([\#9474](matrix-org/synapse#9474)) Synapse 1.28.0rc1 (2021-02-19) ============================== Removal warning --------------- The v1 list accounts API is deprecated and will be removed in a future release. This API was undocumented and misleading. It can be replaced by the [v2 list accounts API](https://github.com/matrix-org/synapse/blob/release-v1.28.0/docs/admin_api/user_admin_api.rst#list-accounts), which has been available since Synapse 1.7.0 (2019-12-13). Please check if you're using any scripts which use the admin API and replace `GET /_synapse/admin/v1/users/<user_id>` with `GET /_synapse/admin/v2/users`. Features -------- - New admin API to get the context of an event: `/_synapse/admin/rooms/{roomId}/context/{eventId}`. ([\#9150](matrix-org/synapse#9150)) - Further improvements to the user experience of registration via single sign-on. ([\#9300](matrix-org/synapse#9300), [\#9301](matrix-org/synapse#9301)) - Add hook to spam checker modules that allow checking file uploads and remote downloads. ([\#9311](matrix-org/synapse#9311)) - Add support for receiving OpenID Connect authentication responses via form `POST`s rather than `GET`s. ([\#9376](matrix-org/synapse#9376)) - Add the shadow-banning status to the admin API for user info. ([\#9400](matrix-org/synapse#9400)) Bugfixes -------- - Fix long-standing bug where sending email notifications would fail for rooms that the server had since left. ([\#9257](matrix-org/synapse#9257)) - Fix bug introduced in Synapse 1.27.0rc1 which meant the "session expired" error page during SSO registration was badly formatted. ([\#9296](matrix-org/synapse#9296)) - Assert a maximum length for some parameters for spec compliance. ([\#9321](matrix-org/synapse#9321), [\#9393](matrix-org/synapse#9393)) - Fix additional errors when previewing URLs: "AttributeError 'NoneType' object has no attribute 'xpath'" and "ValueError: Unicode strings with encoding declaration are not supported. Please use bytes input or XML fragments without declaration.". ([\#9333](matrix-org/synapse#9333)) - Fix a bug causing Synapse to impose the wrong type constraints on fields when processing responses from appservices to `/_matrix/app/v1/thirdparty/user/{protocol}`. ([\#9361](matrix-org/synapse#9361)) - Fix bug where Synapse would occasionally stop reconnecting to Redis after the connection was lost. ([\#9391](matrix-org/synapse#9391)) - Fix a long-standing bug when upgrading a room: "TypeError: '>' not supported between instances of 'NoneType' and 'int'". ([\#9395](matrix-org/synapse#9395)) - Reduce the amount of memory used when generating the URL preview of a file that is larger than the `max_spider_size`. ([\#9421](matrix-org/synapse#9421)) - Fix a long-standing bug in the deduplication of old presence, resulting in no deduplication. ([\#9425](matrix-org/synapse#9425)) - The `ui_auth.session_timeout` config option can now be specified in terms of number of seconds/minutes/etc/. Contributed by Rishabh Arya. ([\#9426](matrix-org/synapse#9426)) - Fix a bug introduced in v1.27.0: "TypeError: int() argument must be a string, a bytes-like object or a number, not 'NoneType." related to the user directory. ([\#9428](matrix-org/synapse#9428)) Updates to the Docker image --------------------------- - Drop support for ARMv7 in Docker images. ([\#9433](matrix-org/synapse#9433)) Improved Documentation ---------------------- - Reorganize CHANGELOG.md. ([\#9281](matrix-org/synapse#9281)) - Add note to `auto_join_rooms` config option explaining existing rooms must be publicly joinable. ([\#9291](matrix-org/synapse#9291)) - Correct name of Synapse's service file in TURN howto. ([\#9308](matrix-org/synapse#9308)) - Fix the braces in the `oidc_providers` section of the sample config. ([\#9317](matrix-org/synapse#9317)) - Update installation instructions on Fedora. ([\#9322](matrix-org/synapse#9322)) - Add HTTP/2 support to the nginx example configuration. Contributed by David Vo. ([\#9390](matrix-org/synapse#9390)) - Update docs for using Gitea as OpenID provider. ([\#9404](matrix-org/synapse#9404)) - Document that pusher instances are shardable. ([\#9407](matrix-org/synapse#9407)) - Fix erroneous documentation from v1.27.0 about updating the SAML2 callback URL. ([\#9434](matrix-org/synapse#9434)) Deprecations and Removals ------------------------- - Deprecate old admin API `GET /_synapse/admin/v1/users/<user_id>`. ([\#9429](matrix-org/synapse#9429)) Internal Changes ---------------- - Fix 'object name reserved for internal use' errors with recent versions of SQLite. ([\#9003](matrix-org/synapse#9003)) - Add experimental support for running Synapse with PyPy. ([\#9123](matrix-org/synapse#9123)) - Deny access to additional IP addresses by default. ([\#9240](matrix-org/synapse#9240)) - Update the `Cursor` type hints to better match PEP 249. ([\#9299](matrix-org/synapse#9299)) - Add debug logging for SRV lookups. Contributed by @Bubu. ([\#9305](matrix-org/synapse#9305)) - Improve logging for OIDC login flow. ([\#9307](matrix-org/synapse#9307)) - Share the code for handling required attributes between the CAS and SAML handlers. ([\#9326](matrix-org/synapse#9326)) - Clean up the code to load the metadata for OpenID Connect identity providers. ([\#9362](matrix-org/synapse#9362)) - Convert tests to use `HomeserverTestCase`. ([\#9377](matrix-org/synapse#9377), [\#9396](matrix-org/synapse#9396)) - Update the version of black used to 20.8b1. ([\#9381](matrix-org/synapse#9381)) - Allow OIDC config to override discovered values. ([\#9384](matrix-org/synapse#9384)) - Remove some dead code from the acceptance of room invites path. ([\#9394](matrix-org/synapse#9394)) - Clean up an unused method in the presence handler code. ([\#9408](matrix-org/synapse#9408)) Synapse 1.27.0 (2021-02-16) =========================== Note that this release includes a change in Synapse to use Redis as a cache ─ as well as a pub/sub mechanism ─ if Redis support is enabled for workers. No action is needed by server administrators, and we do not expect resource usage of the Redis instance to change dramatically. This release also changes the callback URI for OpenID Connect (OIDC) and SAML2 identity providers. If your server is configured to use single sign-on via an OIDC/OAuth2 or SAML2 IdP, you may need to make configuration changes. Please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. This release also changes escaping of variables in the HTML templates for SSO or email notifications. If you have customised these templates, please review [UPGRADE.rst](UPGRADE.rst) for more details on these changes. Bugfixes -------- - Fix building Docker images for armv7. ([\#9405](matrix-org/synapse#9405)) Synapse 1.27.0rc2 (2021-02-11) ============================== Features -------- - Further improvements to the user experience of registration via single sign-on. ([\#9297](matrix-org/synapse#9297)) Bugfixes -------- - Fix ratelimiting introduced in v1.27.0rc1 for invites to respect the `ratelimit` flag on application services. ([\#9302](matrix-org/synapse#9302)) - Do not automatically calculate `public_baseurl` since it can be wrong in some situations. Reverts behaviour introduced in v1.26.0. ([\#9313](matrix-org/synapse#9313)) Improved Documentation ---------------------- - Clarify the sample configuration for changes made to the template loading code. ([\#9310](matrix-org/synapse#9310)) Synapse 1.27.0rc1 (2021-02-02) ============================== Features -------- - Add an admin API for getting and deleting forward extremities for a room. ([\#9062](matrix-org/synapse#9062)) - Add an admin API for retrieving the current room state of a room. ([\#9168](matrix-org/synapse#9168)) - Add experimental support for allowing clients to pick an SSO Identity Provider ([MSC2858](matrix-org/matrix-spec-proposals#2858)). ([\#9183](matrix-org/synapse#9183), [\#9242](matrix-org/synapse#9242)) - Add an admin API endpoint for shadow-banning users. ([\#9209](matrix-org/synapse#9209)) - Add ratelimits to the 3PID `/requestToken` APIs. ([\#9238](matrix-org/synapse#9238)) - Add support to the OpenID Connect integration for adding the user's email address. ([\#9245](matrix-org/synapse#9245)) - Add ratelimits to invites in rooms and to specific users. ([\#9258](matrix-org/synapse#9258)) - Improve the user experience of setting up an account via single-sign on. ([\#9262](matrix-org/synapse#9262), [\#9272](matrix-org/synapse#9272), [\#9275](matrix-org/synapse#9275), [\#9276](matrix-org/synapse#9276), [\#9277](matrix-org/synapse#9277), [\#9286](matrix-org/synapse#9286), [\#9287](matrix-org/synapse#9287)) - Add phone home stats for encrypted messages. ([\#9283](matrix-org/synapse#9283)) - Update the redirect URI for OIDC authentication. ([\#9288](matrix-org/synapse#9288)) Bugfixes -------- - Fix spurious errors in logs when deleting a non-existant pusher. ([\#9121](matrix-org/synapse#9121)) - Fix a long-standing bug where Synapse would return a 500 error when a thumbnail did not exist (and auto-generation of thumbnails was not enabled). ([\#9163](matrix-org/synapse#9163)) - Fix a long-standing bug where an internal server error was raised when attempting to preview an HTML document in an unknown character encoding. ([\#9164](matrix-org/synapse#9164)) - Fix a long-standing bug where invalid data could cause errors when calculating the presentable room name for push. ([\#9165](matrix-org/synapse#9165)) - Fix bug where we sometimes didn't detect that Redis connections had died, causing workers to not see new data. ([\#9218](matrix-org/synapse#9218)) - Fix a bug where `None` was passed to Synapse modules instead of an empty dictionary if an empty module `config` block was provided in the homeserver config. ([\#9229](matrix-org/synapse#9229)) - Fix a bug in the `make_room_admin` admin API where it failed if the admin with the greatest power level was not in the room. Contributed by Pankaj Yadav. ([\#9235](matrix-org/synapse#9235)) - Prevent password hashes from getting dropped if a client failed threepid validation during a User Interactive Auth stage. Removes a workaround for an ancient bug in Riot Web <v0.7.4. ([\#9265](matrix-org/synapse#9265)) - Fix single-sign-on when the endpoints are routed to synapse workers. ([\#9271](matrix-org/synapse#9271)) Improved Documentation ---------------------- - Add docs for using Gitea as OpenID provider. ([\#9134](matrix-org/synapse#9134)) - Add link to Matrix VoIP tester for turn-howto. ([\#9135](matrix-org/synapse#9135)) - Add notes on integrating with Facebook for SSO login. ([\#9244](matrix-org/synapse#9244)) Deprecations and Removals ------------------------- - The `service_url` parameter in `cas_config` is deprecated in favor of `public_baseurl`. ([\#9199](matrix-org/synapse#9199)) - Add new endpoint `/_synapse/client/saml2` for SAML2 authentication callbacks, and deprecate the old endpoint `/_matrix/saml2`. ([\#9289](matrix-org/synapse#9289)) Internal Changes ---------------- - Add tests to `test_user.UsersListTestCase` for List Users Admin API. ([\#9045](matrix-org/synapse#9045)) - Various improvements to the federation client. ([\#9129](matrix-org/synapse#9129)) - Speed up chain cover calculation when persisting a batch of state events at once. ([\#9176](matrix-org/synapse#9176)) - Add a `long_description_type` to the package metadata. ([\#9180](matrix-org/synapse#9180)) - Speed up batch insertion when using PostgreSQL. ([\#9181](matrix-org/synapse#9181), [\#9188](matrix-org/synapse#9188)) - Emit an error at startup if different Identity Providers are configured with the same `idp_id`. ([\#9184](matrix-org/synapse#9184)) - Improve performance of concurrent use of `StreamIDGenerators`. ([\#9190](matrix-org/synapse#9190)) - Add some missing source directories to the automatic linting script. ([\#9191](matrix-org/synapse#9191)) - Precompute joined hosts and store in Redis. ([\#9198](matrix-org/synapse#9198), [\#9227](matrix-org/synapse#9227)) - Clean-up template loading code. ([\#9200](matrix-org/synapse#9200)) - Fix the Python 3.5 old dependencies build. ([\#9217](matrix-org/synapse#9217)) - Update `isort` to v5.7.0 to bypass a bug where it would disagree with `black` about formatting. ([\#9222](matrix-org/synapse#9222)) - Add type hints to handlers code. ([\#9223](matrix-org/synapse#9223), [\#9232](matrix-org/synapse#9232)) - Fix Debian package building on Ubuntu 16.04 LTS (Xenial). ([\#9254](matrix-org/synapse#9254)) - Minor performance improvement during TLS handshake. ([\#9255](matrix-org/synapse#9255)) - Refactor the generation of summary text for email notifications. ([\#9260](matrix-org/synapse#9260)) - Restore PyPy compatibility by not calling CPython-specific GC methods when under PyPy. ([\#9270](matrix-org/synapse#9270))

erikjohnston added 5 commits January 21, 2021 19:30

Update typing stubs

4c56124

Add an ExternalCache

baac88a

Use external cache to precomputed joined hosts

62857fb

Add some metrics

007c326

Newsfie

e980279

erikjohnston force-pushed the erikj/cache_joined_hosts_redis branch from 0255c47 to e980279 Compare January 21, 2021 19:30

Handle external cache redis connection in unit tests

eacd623

erikjohnston requested a review from a team January 22, 2021 11:33

Don't require a redis connection in tests

da4ad62

clokep reviewed Jan 22, 2021

View reviewed changes

synapse/federation/sender/__init__.py Outdated Show resolved Hide resolved

erikjohnston added 3 commits January 22, 2021 15:58

Add an ExternalCache.is_enabled

dfb92dd

Use milliseconds for expiry time

a5c3589

Remove unused attribute

51b9cb3

erikjohnston requested a review from clokep January 22, 2021 16:09

clokep suggested changes Jan 22, 2021

View reviewed changes

s/set_cache/set/

a524c1b

erikjohnston requested a review from clokep January 22, 2021 22:47

richvdh reviewed Jan 25, 2021

View reviewed changes

synapse/replication/tcp/external_cache.py Outdated Show resolved Hide resolved

erikjohnston added 2 commits January 25, 2021 16:05

Remove spurious comment

0125d67

Correctly handle log contexts when calling redis api

f37f0e2

richvdh reviewed Jan 25, 2021

View reviewed changes

clokep removed their request for review January 25, 2021 16:45

clokep mentioned this pull request Jan 25, 2021

Periodically send pings to detect dead Redis connections #9218

Merged

erikjohnston added 4 commits January 26, 2021 11:02

Merge remote-tracking branch 'origin/develop' into erikj/cache_joined…

7a4fbb5

…_hosts_redis

Share one outbound redis connection

b8f3ea4

Don't check cache for create events

77764e0

Precalculate joined hosts for events generated by federation too

6fd1172

Cache the generated state group in the state entry

c5cc378

This ensures that when we call `resolve_state_groups` again we get the assigned state group returned.

erikjohnston requested a review from a team January 26, 2021 12:00

s/expiry_ms/expiry_ms/

e78ea69

clokep approved these changes Jan 26, 2021

View reviewed changes

synapse/handlers/message.py Outdated Show resolved Hide resolved

Make func public and fix docstring

2103b32

erikjohnston merged commit dd8da8c into develop Jan 26, 2021

erikjohnston deleted the erikj/cache_joined_hosts_redis branch January 26, 2021 13:57

clokep mentioned this pull request Jan 26, 2021

Federation senders fall behind after restart #9172

Closed

erikjohnston mentioned this pull request Jan 26, 2021

Add a note to changelog about redis usage #9227

Merged

clokep mentioned this pull request Jan 28, 2021

Large amounts of repeated DB queries when using multiple federation sender workers #9113

Closed

clokep mentioned this pull request Dec 2, 2021

Using an external cache server #2123

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Precompute joined hosts and store in Redis #9198

Precompute joined hosts and store in Redis #9198

erikjohnston commented Jan 21, 2021

clokep Jan 22, 2021

clokep Jan 22, 2021

erikjohnston Jan 22, 2021

clokep Jan 22, 2021

erikjohnston Jan 22, 2021

clokep Jan 22, 2021

clokep Jan 22, 2021

erikjohnston Jan 22, 2021

clokep commented Jan 22, 2021

clokep commented Jan 22, 2021

clokep left a comment

richvdh Jan 25, 2021

erikjohnston Jan 25, 2021

erikjohnston Jan 26, 2021

erikjohnston commented Jan 26, 2021

clokep left a comment

erikjohnston commented Jan 26, 2021

Precompute joined hosts and store in Redis #9198

Precompute joined hosts and store in Redis #9198

Conversation

erikjohnston commented Jan 21, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep commented Jan 22, 2021

clokep commented Jan 22, 2021

clokep left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

erikjohnston commented Jan 26, 2021

clokep left a comment

Choose a reason for hiding this comment

erikjohnston commented Jan 26, 2021