This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Track DB txn times w/ two counters, not histogram #13342
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
added 2 commits
July 20, 2022 18:23
By my reckoning, this should mean that each txn `desc` produces 2 time series, down from 17. If we don't like this, we could keep the histogram but configure it with coarser buckets. Closes #11081.
DMRobertson
force-pushed
the
dmr/txn_metrics
branch
from
114c768 to
490af3a
Compare
DMRobertson
commented
Jul 20, 2022
| @@ -0,0 +1 @@ | |||
| When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. | |||
There was a problem hiding this comment.
Unsure if this should be .feature?
erikjohnston
approved these changes
Jul 21, 2022
Alladin9393
added a commit
to BitorbitLabs/synapse
that referenced
this pull request
Aug 12, 2022
Synapse 1.64.0 (2022-08-02) =========================== No significant changes since 1.64.0rc2. Deprecation Warning ------------------- Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. [Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email) Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406)) Synapse 1.64.0rc1 (2022-07-26) ============================== This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu. Features -------- - Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038)) - Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205)) - Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208)) - Add support for room version 10. ([\matrix-org#13220](matrix-org#13220)) - Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276)) - Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317)) Bugfixes -------- - Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263)) - Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270)) - Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278)) - Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296)) - Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350)) - Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352)) Improved Documentation ---------------------- - Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231)) - Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261)) - Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271)) - Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314)) - Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333)) - Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338)) - Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344)) Deprecations and Removals ------------------------- - Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967)) - Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192)) - Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239)) - Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326)) Internal Changes ---------------- - Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942)) - Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943)) - Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094)) - Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172)) - Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175)) - Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198)) - Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215)) - Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218)) - Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224)) - Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240)) - Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308)) - Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251)) - Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257)) - Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258)) - Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260)) - Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266)) - Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274)) - Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279)) - Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307)) - Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285)) - Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292)) - Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297)) - Improve performance of query `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299)) - Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300)) - Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303)) - Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310)) - Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311)) - Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313)) - Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318)) - Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320)) - Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323)) - Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324)) - Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362)) - Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349)) - When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342)) - Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354))
Fizzadar
added a commit
to beeper/synapse-legacy-fork
that referenced
this pull request
Aug 23, 2022
Synapse 1.64.0 (2022-08-02) =========================== No significant changes since 1.64.0rc2. Deprecation Warning ------------------- Synapse v1.66.0 will remove the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. If you require your homeserver to verify e-mail addresses or to support password resets via e-mail, please configure your homeserver with SMTP access so that it can send e-mails on its own behalf. [Consult the configuration documentation for more information.](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#email) Synapse 1.64.0rc2 (2022-07-29) ============================== This RC reintroduces support for `account_threepid_delegates.email`, which was removed in 1.64.0rc1. It remains deprecated and will be removed altogether in Synapse v1.66.0. ([\matrix-org#13406](matrix-org#13406)) Synapse 1.64.0rc1 (2022-07-26) ============================== This RC removed the ability to delegate the tasks of verifying email address ownership, and password reset confirmation, to an identity server. We have also stopped building `.deb` packages for Ubuntu 21.10 as it is no longer an active version of Ubuntu. Features -------- - Improve error messages when media thumbnails cannot be served. ([\matrix-org#13038](matrix-org#13038)) - Allow pagination from remote event after discovering it from [MSC3030](matrix-org/matrix-spec-proposals#3030) `/timestamp_to_event`. ([\matrix-org#13205](matrix-org#13205)) - Add a `room_type` field in the responses for the list room and room details admin APIs. Contributed by @andrewdoh. ([\matrix-org#13208](matrix-org#13208)) - Add support for room version 10. ([\matrix-org#13220](matrix-org#13220)) - Add per-room rate limiting for room joins. For each room, Synapse now monitors the rate of join events in that room, and throttles additional joins if that rate grows too large. ([\matrix-org#13253](matrix-org#13253), [\matrix-org#13254](matrix-org#13254), [\matrix-org#13255](matrix-org#13255), [\matrix-org#13276](matrix-org#13276)) - Support Implicit TLS (TLS without using a STARTTLS upgrade, typically on port 465) for sending emails, enabled by the new option `force_tls`. Contributed by Jan Schär. ([\matrix-org#13317](matrix-org#13317)) Bugfixes -------- - Fix a bug introduced in Synapse 1.15.0 where adding a user through the Synapse Admin API with a phone number would fail if the `enable_email_notifs` and `email_notifs_for_new_users` options were enabled. Contributed by @thomasweston12. ([\matrix-org#13263](matrix-org#13263)) - Fix a bug introduced in Synapse 1.40.0 where a user invited to a restricted room would be briefly unable to join. ([\matrix-org#13270](matrix-org#13270)) - Fix a long-standing bug where, in rare instances, Synapse could store the incorrect state for a room after a state resolution. ([\matrix-org#13278](matrix-org#13278)) - Fix a bug introduced in v1.18.0 where the `synapse_pushers` metric would overcount pushers when they are replaced. ([\matrix-org#13296](matrix-org#13296)) - Disable autocorrection and autocapitalisation on the username text field shown during registration when using SSO. ([\matrix-org#13350](matrix-org#13350)) - Update locked version of `frozendict` to 2.3.3, which has fixes for memory leaks affecting `/sync`. ([\matrix-org#13284](matrix-org#13284), [\matrix-org#13352](matrix-org#13352)) Improved Documentation ---------------------- - Provide an example of using the Admin API. Contributed by @jejo86. ([\matrix-org#13231](matrix-org#13231)) - Move the documentation for how URL previews work to the URL preview module. ([\matrix-org#13233](matrix-org#13233), [\matrix-org#13261](matrix-org#13261)) - Add another `contrib` script to help set up worker processes. Contributed by @villepeh. ([\matrix-org#13271](matrix-org#13271)) - Document that certain config options were added or changed in Synapse 1.62. Contributed by @behrmann. ([\matrix-org#13314](matrix-org#13314)) - Document the new `rc_invites.per_issuer` throttling option added in Synapse 1.63. ([\matrix-org#13333](matrix-org#13333)) - Mention that BuildKit is needed when building Docker images for tests. ([\matrix-org#13338](matrix-org#13338)) - Improve Caddy reverse proxy documentation. ([\matrix-org#13344](matrix-org#13344)) Deprecations and Removals ------------------------- - Drop tables that were formerly used for groups/communities. ([\matrix-org#12967](matrix-org#12967)) - Drop support for delegating email verification to an external server. ([\matrix-org#13192](matrix-org#13192)) - Drop support for calling `/_matrix/client/v3/account/3pid/bind` without an `id_access_token`, which was not permitted by the spec. Contributed by @Vetchu. ([\matrix-org#13239](matrix-org#13239)) - Stop building `.deb` packages for Ubuntu 21.10 (Impish Indri), which has reached end of life. ([\matrix-org#13326](matrix-org#13326)) Internal Changes ---------------- - Use lower transaction isolation level when purging rooms to avoid serialization errors. Contributed by Nick @ Beeper. ([\matrix-org#12942](matrix-org#12942)) - Remove code which incorrectly attempted to reconcile state with remote servers when processing incoming events. ([\matrix-org#12943](matrix-org#12943)) - Make the AS login method call `Auth.get_user_by_req` for checking the AS token. ([\matrix-org#13094](matrix-org#13094)) - Always use a version of canonicaljson that supports the C implementation of frozendict. ([\matrix-org#13172](matrix-org#13172)) - Add prometheus counters for ephemeral events and to device messages pushed to app services. Contributed by Brad @ Beeper. ([\matrix-org#13175](matrix-org#13175)) - Refactor receipts servlet logic to avoid duplicated code. ([\matrix-org#13198](matrix-org#13198)) - Preparation for database schema simplifications: populate `state_key` and `rejection_reason` for existing rows in the `events` table. ([\matrix-org#13215](matrix-org#13215)) - Remove unused database table `event_reference_hashes`. ([\matrix-org#13218](matrix-org#13218)) - Further reduce queries used sending events when creating new rooms. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13224](matrix-org#13224)) - Call the v2 identity service `/3pid/unbind` endpoint, rather than v1. Contributed by @Vetchu. ([\matrix-org#13240](matrix-org#13240)) - Use an asynchronous cache wrapper for the get event cache. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13242](matrix-org#13242), [\matrix-org#13308](matrix-org#13308)) - Optimise federation sender and appservice pusher event stream processing queries. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13251](matrix-org#13251)) - Log the stack when waiting for an entire room to be un-partial stated. ([\matrix-org#13257](matrix-org#13257)) - Fix spurious warning when fetching state after a missing prev event. ([\matrix-org#13258](matrix-org#13258)) - Clean-up tests for notifications. ([\matrix-org#13260](matrix-org#13260)) - Do not fail build if complement with workers fails. ([\matrix-org#13266](matrix-org#13266)) - Don't pull out state in `compute_event_context` for unconflicted state. ([\matrix-org#13267](matrix-org#13267), [\matrix-org#13274](matrix-org#13274)) - Reduce the rebuild time for the complement-synapse docker image. ([\matrix-org#13279](matrix-org#13279)) - Don't pull out the full state when creating an event. ([\matrix-org#13281](matrix-org#13281), [\matrix-org#13307](matrix-org#13307)) - Upgrade from Poetry 1.1.12 to 1.1.14, to fix bugs when locking packages. ([\matrix-org#13285](matrix-org#13285)) - Make `DictionaryCache` expire full entries if they haven't been queried in a while, even if specific keys have been queried recently. ([\matrix-org#13292](matrix-org#13292)) - Use `HTTPStatus` constants in place of literals in tests. ([\matrix-org#13297](matrix-org#13297)) - Improve performance of query `_get_subset_users_in_room_with_profiles`. ([\matrix-org#13299](matrix-org#13299)) - Up batch size of `bulk_get_push_rules` and `_get_joined_profiles_from_event_ids`. ([\matrix-org#13300](matrix-org#13300)) - Remove unnecessary `json.dumps` from tests. ([\matrix-org#13303](matrix-org#13303)) - Reduce memory usage of sending dummy events. ([\matrix-org#13310](matrix-org#13310)) - Prevent formatting changes of [matrix-org#3679](matrix-org#3679) from appearing in `git blame`. ([\matrix-org#13311](matrix-org#13311)) - Change `get_users_in_room` and `get_rooms_for_user` caches to enable pruning of old entries. ([\matrix-org#13313](matrix-org#13313)) - Validate federation destinations and log an error if a destination is invalid. ([\matrix-org#13318](matrix-org#13318)) - Fix `FederationClient.get_pdu()` returning events from the cache as `outliers` instead of original events we saw over federation. ([\matrix-org#13320](matrix-org#13320)) - Reduce memory usage of state caches. ([\matrix-org#13323](matrix-org#13323)) - Reduce the amount of state we store in the `state_cache`. ([\matrix-org#13324](matrix-org#13324)) - Add missing type hints to open tracing module. ([\matrix-org#13328](matrix-org#13328), [\matrix-org#13345](matrix-org#13345), [\matrix-org#13362](matrix-org#13362)) - Remove old base slaved store and de-duplicate cache ID generators. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13329](matrix-org#13329), [\matrix-org#13349](matrix-org#13349)) - When reporting metrics is enabled, use ~8x less data to describe DB transaction metrics. ([\matrix-org#13342](matrix-org#13342)) - Faster room joins: skip soft fail checks while Synapse only has partial room state, since the current membership of event senders may not be accurately known. ([\matrix-org#13354](matrix-org#13354)) # -----BEGIN PGP SIGNATURE----- # # iQIzBAABCgAdFiEE8SRSDO7gYkSP4chELS76LzL74EcFAmLo+zIACgkQLS76LzL7 # 4EehbRAAronXZtWM+ViMxPsiDj70KXYOKK117pGXK5XGf3Tyqb/vExA7c7bfimyW # d3FW855fe27AMsSfcMGDpxhggVa8sZDSdvQumt5jqDXrzC348mW/FYtgcYOxkoIa # Hh2/7V26CxWFsv8eVF3hwpualelT9lp2sedWXCQtdAkcQoWs2JwBsnoxSDliDZHg # jc4mBFBAkah5CJ3bcZuZXRsr9doKxDOAXUv19RXhdwEGO82mpSbwQ8P0mcw2S8zr # aAVza7jkVAza6ahg9qE0lMpi8uYE9/mt5JBnfrv/JxC7ZZfBg9jyHKaxFrzpjFsj # 3g0jhqzcNxRskD1sk1GKGVy7D9oTg1WVpii5l3M93KguSDLKxomouhgekWOxMPBe # 43xVdDI13ohsex+1QBnGnTSP7jZcfODnfvzSdyHQv6ef4k+OplRdfMA0QjkUcI5j # ocJlkm2D02vw1mnU3hHNdw9ri3vkaS1Qwfsz3ZEYgn6OcZOeKAWn351WMXF/F1fm # HYeQ5uMud+i+EekBtR8Op9ZICHt9Ogp49172enlSGzeyeD3yUk5HMAMrzJfmsp3W # /LCCONkRrV+R8TRByUQE9YtqxUgn+eSgB5Ew/2C/WB54pZHtco+rPqkY1Bhan4QJ # LeZTuzDKeXzgho1D5b4quEC2AWAqz3GeIvEVuOZCt8rJoMMRslg= # =RRRX # -----END PGP SIGNATURE----- # gpg: Signature made Tue Aug 2 11:23:46 2022 BST # gpg: using RSA key F124520CEEE062448FE1C8442D2EFA2F32FBE047 # gpg: Can't check signature: No public key # Conflicts: # synapse/rest/client/read_marker.py # synapse/rest/client/receipts.py # synapse/storage/databases/main/events_worker.py # synapse/storage/databases/main/purge_events.py # tests/rest/client/test_rooms.py # tests/storage/test_event_push_actions.py
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
By my reckoning, this should mean that each txn
descproduces 2 timeseries, down from 17. If we don't like this, we could keep the histogram
but configure it with coarser buckets.
This saves on storage and CPU cycles. We don't use the full histogram distribution in our dashboards, so we think this is safe to drop.
Closes #11081.