Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Update locked frozendict version to 2.3.3 #13352

Merged
merged 3 commits into from
Jul 22, 2022

Conversation

squahtx
Copy link
Contributor

@squahtx squahtx commented Jul 21, 2022

frozendict 2.3.3 includes fixes for memory leaks that get triggered
during /sync.

Resolves #13282.
Resolves #13286.

Signed-off-by: Sean Quah seanq@matrix.org

frozendict 2.3.3 includes fixes for memory leaks that get triggered
during `/sync`.

Signed-off-by: Sean Quah <seanq@matrix.org>
@squahtx squahtx requested a review from a team as a code owner July 21, 2022 17:37
Copy link
Contributor

@DMRobertson DMRobertson left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM. The only thing I'm not sure about: should we call this out in the upgrade notes, for people who are installing from pip---probably not?

(I'm assuming we can't bump the dependency lower bounds because it'll cause packager pain.)

@clokep
Copy link
Member

clokep commented Jul 21, 2022

LGTM. The only thing I'm not sure about: should we call this out in the upgrade notes, for people who are installing from pip---probably not?

I thought that we've done this in the past, but I can't find examples except for security advisories (https://github.com/matrix-org/synapse/blob/develop/docs/changelogs/CHANGES-2020.md#security-advisory, https://github.com/matrix-org/synapse/blob/master/CHANGES.md#synapse-1520-2022-02-08).

#7176 (comment) is the most similar thing I can see, where we didn't make any changes.

We also could call it out in the blog post, not sure if that makes more sense.

@DMRobertson
Copy link
Contributor

For cross-referencing, dealing with bugs in dependencies falls under #13315.

@squahtx squahtx requested a review from DMRobertson July 21, 2022 18:43
@squahtx
Copy link
Contributor Author

squahtx commented Jul 21, 2022

@DMRobertson I've added an upgrade note. Let me know if it looks alright!

@squahtx squahtx requested a review from a team July 21, 2022 18:47
@squahtx
Copy link
Contributor Author

squahtx commented Jul 21, 2022

Just remembered that dmr is out.

@matrix-org/synapse-core A quick review of the release note would be appreciated!

@erikjohnston erikjohnston merged commit 0fa41a7 into develop Jul 22, 2022
@erikjohnston erikjohnston deleted the squah/update_frozendict_to_2_3_3 branch July 22, 2022 09:26
Alladin9393 added a commit to BitorbitLabs/synapse that referenced this pull request Aug 12, 2022
Synapse 1.64.0 (2022-08-02)
===========================

No significant changes since 1.64.0rc2.

Deprecation Warning
-------------------

Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)

Synapse 1.64.0rc2 (2022-07-29)
==============================

This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406))

Synapse 1.64.0rc1 (2022-07-26)
==============================

This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu.

Features
--------

- Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038))
- Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205))
- Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208))
- Add support for room version 10. ([\matrix-org#13220](matrix-org#13220))
- Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276))
- Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263))
- Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270))
- Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278))
- Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296))
- Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350))
- Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352))

Improved Documentation
----------------------

- Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231))
- Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261))
- Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271))
- Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314))
- Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333))
- Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338))
- Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344))

Deprecations and Removals
-------------------------

- Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967))
- Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192))
- Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239))
- Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326))

Internal Changes
----------------

- Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942))
- Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943))
- Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094))
- Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172))
- Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175))
- Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198))
- Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215))
- Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218))
- Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224))
- Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240))
- Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308))
- Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251))
- Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257))
- Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258))
- Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260))
- Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266))
- Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274))
- Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279))
- Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307))
- Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285))
- Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292))
- Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297))
- Improve performance of query  `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299))
- Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300))
- Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303))
- Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310))
- Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311))
- Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313))
- Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318))
- Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320))
- Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323))
- Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324))
- Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362))
- Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349))
- When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342))
- Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Aug 23, 2022
Synapse 1.64.0 (2022-08-02)
===========================

No significant changes since 1.64.0rc2.

Deprecation Warning
-------------------

Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf.
[Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email)

Synapse 1.64.0rc2 (2022-07-29)
==============================

This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406))

Synapse 1.64.0rc1 (2022-07-26)
==============================

This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server.

We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu.

Features
--------

- Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038))
- Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205))
- Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208))
- Add support for room version 10. ([\matrix-org#13220](matrix-org#13220))
- Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276))
- Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263))
- Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270))
- Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278))
- Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296))
- Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350))
- Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352))

Improved Documentation
----------------------

- Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231))
- Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261))
- Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271))
- Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314))
- Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333))
- Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338))
- Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344))

Deprecations and Removals
-------------------------

- Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967))
- Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192))
- Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239))
- Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326))

Internal Changes
----------------

- Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942))
- Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943))
- Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094))
- Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172))
- Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175))
- Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198))
- Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215))
- Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218))
- Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224))
- Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240))
- Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308))
- Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251))
- Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257))
- Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258))
- Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260))
- Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266))
- Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274))
- Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279))
- Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307))
- Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285))
- Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292))
- Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297))
- Improve performance of query  `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299))
- Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300))
- Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303))
- Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310))
- Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311))
- Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313))
- Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318))
- Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320))
- Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323))
- Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324))
- Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362))
- Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349))
- When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342))
- Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))

# -----BEGIN PGP SIGNATURE-----
#
# iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmLo+zIACgkQLS76LzL7
# 4EehbRAAronXZtWM+ViMxPsiDj70KXYOKK117pGXK5XGf3Tyqb/vExA7c7bfimyW
# d3FW855fe27AMsSfcMGDpxhggVa8sZDSdvQumt5jqDXrzC348mW/FYtgcYOxkoIa
# Hh2/7V26CxWFsv8eVF3hwpualelT9lp2sedWXCQtdAkcQoWs2JwBsnoxSDliDZHg
# jc4mBFBAkah5CJ3bcZuZXRsr9doKxDOAXUv19RXhdwEGO82mpSbwQ8P0mcw2S8zr
# aAVza7jkVAza6ahg9qE0lMpi8uYE9/mt5JBnfrv/JxC7ZZfBg9jyHKaxFrzpjFsj
# 3g0jhqzcNxRskD1sk1GKGVy7D9oTg1WVpii5l3M93KguSDLKxomouhgekWOxMPBe
# 43xVdDI13ohsex+1QBnGnTSP7jZcfODnfvzSdyHQv6ef4k+OplRdfMA0QjkUcI5j
# ocJlkm2D02vw1mnU3hHNdw9ri3vkaS1Qwfsz3ZEYgn6OcZOeKAWn351WMXF/F1fm
# HYeQ5uMud+i+EekBtR8Op9ZICHt9Ogp49172enlSGzeyeD3yUk5HMAMrzJfmsp3W
# /LCCONkRrV+R8TRByUQE9YtqxUgn+eSgB5Ew/2C/WB54pZHtco+rPqkY1Bhan4QJ
# LeZTuzDKeXzgho1D5b4quEC2AWAqz3GeIvEVuOZCt8rJoMMRslg=
# =RRRX
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Aug  2 11:23:46 2022 BST
# gpg:                using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047
# gpg: Can't check signature: No public key

# Conflicts:
#	synapse/rest/client/read_marker.py
#	synapse/rest/client/receipts.py
#	synapse/storage/databases/main/events_worker.py
#	synapse/storage/databases/main/purge_events.py
#	tests/rest/client/test_rooms.py
#	tests/storage/test_event_push_actions.py
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
4 participants