This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
-
-
Notifications
You must be signed in to change notification settings - Fork 2.1k
Refactor presence so we can prune user in room caches #13313
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
erikjohnston
force-pushed
the
erikj/prune_unread_rooms
branch
2 times, most recently
from
July 18, 2022 13:54
69ac33e
to
8f9e1b6
Compare
erikjohnston
force-pushed
the
erikj/prune_unread_rooms
branch
from
July 22, 2022 10:01
8f9e1b6
to
49757eb
Compare
squahtx
approved these changes
Jul 22, 2022
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
The refactor looks equivalent. It's hard to tell how the new cache will perform, since it'll be cleared more often on bigger homeservers. It sounds like it's fine in practice, so I'm happy.
A few minor typos:
) -> Set[str]: | ||
"""Return the set of users who share a room with the first users""" |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Suggested change
"""Return the set of users who share a room with the first users""" | |
"""Return the subset of users who share a room with the first user""" |
Alladin9393
added a commit
to BitorbitLabs/synapse
that referenced
this pull request
Aug 12, 2022
Synapse 1.64.0 (2022-08-02) =========================== No significant changes since 1.64.0rc2. Deprecation Warning ------------------- Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. [Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email) Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406)) Synapse 1.64.0rc1 (2022-07-26) ============================== This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu. Features -------- - Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038)) - Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205)) - Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208)) - Add support for room version 10. ([\matrix-org#13220](matrix-org#13220)) - Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276)) - Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317)) Bugfixes -------- - Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263)) - Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270)) - Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278)) - Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296)) - Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350)) - Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352)) Improved Documentation ---------------------- - Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231)) - Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261)) - Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271)) - Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314)) - Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333)) - Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338)) - Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344)) Deprecations and Removals ------------------------- - Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967)) - Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192)) - Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239)) - Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326)) Internal Changes ---------------- - Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942)) - Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943)) - Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094)) - Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172)) - Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175)) - Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198)) - Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215)) - Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218)) - Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224)) - Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240)) - Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308)) - Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251)) - Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257)) - Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258)) - Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260)) - Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266)) - Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274)) - Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279)) - Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307)) - Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285)) - Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292)) - Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297)) - Improve performance of query `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299)) - Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300)) - Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303)) - Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310)) - Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311)) - Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313)) - Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318)) - Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320)) - Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323)) - Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324)) - Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362)) - Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349)) - When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342)) - Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))
Fizzadar
added a commit
to beeper/synapse-legacy-fork
that referenced
this pull request
Aug 23, 2022
Synapse 1.64.0 (2022-08-02) =========================== No significant changes since 1.64.0rc2. Deprecation Warning ------------------- Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. [Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email) Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406)) Synapse 1.64.0rc1 (2022-07-26) ============================== This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu. Features -------- - Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038)) - Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205)) - Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208)) - Add support for room version 10. ([\matrix-org#13220](matrix-org#13220)) - Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276)) - Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317)) Bugfixes -------- - Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263)) - Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270)) - Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278)) - Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296)) - Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350)) - Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352)) Improved Documentation ---------------------- - Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231)) - Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261)) - Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271)) - Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314)) - Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333)) - Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338)) - Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344)) Deprecations and Removals ------------------------- - Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967)) - Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192)) - Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239)) - Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326)) Internal Changes ---------------- - Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942)) - Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943)) - Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094)) - Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172)) - Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175)) - Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198)) - Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215)) - Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218)) - Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224)) - Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240)) - Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308)) - Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251)) - Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257)) - Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258)) - Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260)) - Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266)) - Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274)) - Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279)) - Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307)) - Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285)) - Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292)) - Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297)) - Improve performance of query `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299)) - Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300)) - Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303)) - Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310)) - Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311)) - Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313)) - Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318)) - Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320)) - Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323)) - Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324)) - Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362)) - Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349)) - When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342)) - Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmLo+zIACgkQLS76LzL7 # 4EehbRAAronXZtWM+ViMxPsiDj70KXYOKK117pGXK5XGf3Tyqb/vExA7c7bfimyW # d3FW855fe27AMsSfcMGDpxhggVa8sZDSdvQumt5jqDXrzC348mW/FYtgcYOxkoIa # Hh2/7V26CxWFsv8eVF3hwpualelT9lp2sedWXCQtdAkcQoWs2JwBsnoxSDliDZHg # jc4mBFBAkah5CJ3bcZuZXRsr9doKxDOAXUv19RXhdwEGO82mpSbwQ8P0mcw2S8zr # aAVza7jkVAza6ahg9qE0lMpi8uYE9/mt5JBnfrv/JxC7ZZfBg9jyHKaxFrzpjFsj # 3g0jhqzcNxRskD1sk1GKGVy7D9oTg1WVpii5l3M93KguSDLKxomouhgekWOxMPBe # 43xVdDI13ohsex+1QBnGnTSP7jZcfODnfvzSdyHQv6ef4k+OplRdfMA0QjkUcI5j # ocJlkm2D02vw1mnU3hHNdw9ri3vkaS1Qwfsz3ZEYgn6OcZOeKAWn351WMXF/F1fm # HYeQ5uMud+i+EekBtR8Op9ZICHt9Ogp49172enlSGzeyeD3yUk5HMAMrzJfmsp3W # /LCCONkRrV+R8TRByUQE9YtqxUgn+eSgB5Ew/2C/WB54pZHtco+rPqkY1Bhan4QJ # LeZTuzDKeXzgho1D5b4quEC2AWAqz3GeIvEVuOZCt8rJoMMRslg= # =RRRX # -----END PGP SIGNATURE----- # gpg: Signature made Tue Aug 2 11:23:46 2022 BST # gpg: using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047 # gpg: Can't check signature: No public key # Conflicts: # synapse/rest/client/read_marker.py # synapse/rest/client/receipts.py # synapse/storage/databases/main/events_worker.py # synapse/storage/databases/main/purge_events.py # tests/rest/client/test_rooms.py # tests/storage/test_event_push_actions.py
MatMaul
pushed a commit
that referenced
this pull request
Oct 20, 2022
Signed-off-by: Mathieu Velten <mathieuv@matrix.org>
4 tasks
DMRobertson
pushed a commit
that referenced
this pull request
Oct 27, 2022
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
See #10826 and #10786 for context as to why we had to disable pruning on those caches.
get_users_who_share_room_with_user
is currently only called frequently in the presence code. So we just need to refactor presence to call the function less frequently, and then we can remove the various levels of caching that is going on.Note: I've tested this on jki.re, and it doesn't seem to change the performance much. The DB query also seems fairly efficient based on testing on jki.re and matrix.org DBs.