Sliding sync: Correctly track which read receipts we have or have not sent down. #17575

erikjohnston · 2024-08-15T11:41:34Z

Add connection tracking to the receipts extension.

Based on #17574

MadLittleMods · 2024-08-15T22:49:54Z

synapse/handlers/sliding_sync.py

@@ -3128,23 +3221,28 @@ class PerConnectionState:
        rooms: The status of each room for the events stream.


Update docstring with receipts

receipts hasn't been added to the Attributes in the docstring yet

synapse/handlers/sliding_sync.py

MadLittleMods · 2024-08-15T23:26:26Z

synapse/handlers/sliding_sync.py

+                room_result = actual_room_response_map.get(room_id)
+                if room_result is not None and room_result.initial:
+                    initial_rooms.add(room_id)
+                    continue


My initial gut reaction was that we should just rely on what per_connection_state.receipts says for whether something is an initial room. But I think it makes to try to match the timeline with receipts when all of those timeline messages are available. We should add a comment about why we care to do this.

Since this interacts with initial: true, we should also maybe note that there are going to be weird interactions if people enable/disable the receipts extension during the lifetime of the connection. For example, if they have the receipts extension disabled at first and then enable it later; they'll get the initial: true chunk of timeline but no receipts, and then on incremental sync when the receipts extension is enabled, there won't be many/any timeline events so all of the receipts will be filtered out.

synapse/handlers/sliding_sync.py

synapse/storage/databases/main/receipts.py

tests/rest/client/sliding_sync/test_extensions.py

MadLittleMods · 2024-08-16T00:34:11Z

tests/rest/client/utils.py

+            content={},
+            access_token=tok,
+        )
+        assert channel.code == HTTPStatus.OK, channel.text_body


Should we use this helper for the rest of the receipts tests?

Yup, we should, just didn't want to inflate the PR

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

…rikj/ss_receipts2

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

synapse/handlers/sliding_sync.py

MadLittleMods · 2024-08-19T17:54:10Z

synapse/handlers/sliding_sync.py

@@ -3128,23 +3221,28 @@ class PerConnectionState:
        rooms: The status of each room for the events stream.


receipts hasn't been added to the Attributes in the docstring yet

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

This supersedes #17503, given the per-connection state is being heavily rewritten it felt easier to recreate the PR on top of that work. This correctly handles the case of timeline limits going up and down. This does not handle changes in `required_state`, but that can be done as a separate PR. Based on #17575. --------- Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

@kfiven

# Synapse 1.114.0 (2024-09-02) This release enables support for [MSC4186](matrix-org/matrix-spec-proposals#4186) — Simplified Sliding Sync. This allows using the upcoming releases of the Element X mobile apps without having to run a Sliding Sync Proxy. ### Features - Enable native sliding sync support ([MSC3575](matrix-org/matrix-spec-proposals#3575) and [MSC4186](matrix-org/matrix-spec-proposals#4186)) by default. ([\#17648](element-hq/synapse#17648)) # Synapse 1.114.0rc3 (2024-08-30) ### Bugfixes - Fix regression in v1.114.0rc2 that caused workers to fail to start. ([\#17626](element-hq/synapse#17626)) # Synapse 1.114.0rc2 (2024-08-30) ### Features - Improve cross-signing upload when using [MSC3861](matrix-org/matrix-spec-proposals#3861) to use a custom UIA flow stage, with web fallback support. ([\#17509](element-hq/synapse#17509)) - Make `hash_password` script accept password input from stdin. ([\#17608](element-hq/synapse#17608)) ### Bugfixes - Fix hierarchy returning 403 when room is accessible through federation. Contributed by Krishan (@kfiven). ([\#17194](element-hq/synapse#17194)) - Fix content-length on federation `/thumbnail` responses. ([\#17532](element-hq/synapse#17532)) - Fix authenticated media responses using a wrong limit when following redirects over federation. ([\#17543](element-hq/synapse#17543)) ### Internal Changes - MSC3861: load the issuer and account management URLs from OIDC discovery. ([\#17407](element-hq/synapse#17407)) - Refactor sliding sync class into multiple files. ([\#17595](element-hq/synapse#17595)) - Store sliding sync per-connection state in the database. ([\#17599](element-hq/synapse#17599)) - Make the sliding sync `PerConnectionState` class immutable. ([\#17600](element-hq/synapse#17600)) - Add support to `@tag_args` for standalone functions. ([\#17604](element-hq/synapse#17604)) - Speed up incremental syncs in sliding sync by adding some more caching. ([\#17606](element-hq/synapse#17606)) - Always return the user's own read receipts in sliding sync. ([\#17617](element-hq/synapse#17617)) - Replace `isort` and `black` with `ruff`. ([\#17620](element-hq/synapse#17620)) - Refactor sliding sync code to move room list logic out into a separate class. ([\#17622](element-hq/synapse#17622)) ### Updates to locked dependencies * Bump attrs from 23.2.0 to 24.2.0. ([\#17609](element-hq/synapse#17609)) * Bump cryptography from 42.0.8 to 43.0.0. ([\#17584](element-hq/synapse#17584)) * Bump phonenumbers from 8.13.43 to 8.13.44. ([\#17610](element-hq/synapse#17610)) * Bump pygithub from 2.3.0 to 2.4.0. ([\#17612](element-hq/synapse#17612)) * Bump pyyaml from 6.0.1 to 6.0.2. ([\#17611](element-hq/synapse#17611)) * Bump sentry-sdk from 2.12.0 to 2.13.0. ([\#17585](element-hq/synapse#17585)) * Bump serde from 1.0.206 to 1.0.208. ([\#17581](element-hq/synapse#17581)) * Bump serde from 1.0.208 to 1.0.209. ([\#17613](element-hq/synapse#17613)) * Bump serde_json from 1.0.124 to 1.0.125. ([\#17582](element-hq/synapse#17582)) * Bump serde_json from 1.0.125 to 1.0.127. ([\#17614](element-hq/synapse#17614)) * Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813. ([\#17583](element-hq/synapse#17583)) * Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818. ([\#17586](element-hq/synapse#17586)) # Synapse 1.114.0rc1 (2024-08-20) ### Features - Add a flag to `/versions`, `org.matrix.simplified_msc3575`, to indicate whether experimental sliding sync support has been enabled. ([\#17571](element-hq/synapse#17571)) - Handle changes in `timeline_limit` in experimental sliding sync. ([\#17579](element-hq/synapse#17579)) - Correctly track read receipts that should be sent down in experimental sliding sync. ([\#17575](element-hq/synapse#17575), [\#17589](element-hq/synapse#17589), [\#17592](element-hq/synapse#17592)) ### Bugfixes - Start handlers for new media endpoints when media resource configured. ([\#17483](element-hq/synapse#17483)) - Fix timeline ordering (using `stream_ordering` instead of topological ordering) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17510](element-hq/synapse#17510)) - Fix experimental sliding sync implementation to remember any updates in rooms that were not sent down immediately. ([\#17535](element-hq/synapse#17535)) - Better exclude partially stated rooms if we must await full state in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17538](element-hq/synapse#17538)) - Handle lower-case http headers in `_Mulitpart_Parser_Protocol`. ([\#17545](element-hq/synapse#17545)) - Fix fetching federation signing keys from servers that omit `old_verify_keys`. Contributed by @tulir @ Beeper. ([\#17568](element-hq/synapse#17568)) - Fix bug where we would respond with an error when a remote server asked for media that had a length of 0, using the new multipart federation media endpoint. ([\#17570](element-hq/synapse#17570)) ### Improved Documentation - Clarify default behaviour of the [`auto_accept_invites.worker_to_run_on`](https://element-hq.github.io/synapse/develop/usage/configuration/config_documentation.html#auto-accept-invites) option. ([\#17515](element-hq/synapse#17515)) - Improve docstrings for profile methods. ([\#17559](element-hq/synapse#17559)) ### Internal Changes - Add more tracing to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17514](element-hq/synapse#17514)) - Fixup comment in sliding sync implementation. ([\#17531](element-hq/synapse#17531)) - Replace override of deprecated method `HTTPAdapter.get_connection` with `get_connection_with_tls_context`. ([\#17536](element-hq/synapse#17536)) - Fix performance of device lists in `/key/changes` and sliding sync. ([\#17537](element-hq/synapse#17537), [\#17548](element-hq/synapse#17548)) - Bump setuptools from 67.6.0 to 72.1.0. ([\#17542](element-hq/synapse#17542)) - Add a utility function for generating random event IDs. ([\#17557](element-hq/synapse#17557)) - Speed up responding to media requests. ([\#17558](element-hq/synapse#17558), [\#17561](element-hq/synapse#17561), [\#17564](element-hq/synapse#17564), [\#17566](element-hq/synapse#17566), [\#17567](element-hq/synapse#17567), [\#17569](element-hq/synapse#17569)) - Test github token before running release script steps. ([\#17562](element-hq/synapse#17562)) - Reduce log spam of multipart files. ([\#17563](element-hq/synapse#17563)) - Refactor per-connection state in experimental sliding sync handler. ([\#17574](element-hq/synapse#17574)) - Add histogram metrics for sliding sync processing time. ([\#17593](element-hq/synapse#17593)) ### Updates to locked dependencies * Bump bytes from 1.6.1 to 1.7.1. ([\#17526](element-hq/synapse#17526)) * Bump lxml from 5.2.2 to 5.3.0. ([\#17550](element-hq/synapse#17550)) * Bump phonenumbers from 8.13.42 to 8.13.43. ([\#17551](element-hq/synapse#17551)) * Bump regex from 1.10.5 to 1.10.6. ([\#17527](element-hq/synapse#17527)) * Bump sentry-sdk from 2.10.0 to 2.12.0. ([\#17553](element-hq/synapse#17553)) * Bump serde from 1.0.204 to 1.0.206. ([\#17556](element-hq/synapse#17556)) * Bump serde_json from 1.0.122 to 1.0.124. ([\#17555](element-hq/synapse#17555)) * Bump sigstore/cosign-installer from 3.5.0 to 3.6.0. ([\#17549](element-hq/synapse#17549)) * Bump types-pyyaml from 6.0.12.20240311 to 6.0.12.20240808. ([\#17552](element-hq/synapse#17552)) * Bump types-requests from 2.31.0.20240406 to 2.32.0.20240712. ([\#17524](element-hq/synapse#17524)) # Synapse 1.113.0 (2024-08-13) No significant changes since 1.113.0rc1. # Synapse 1.113.0rc1 (2024-08-06) ### Features - Track which rooms have been sent to clients in the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17447](element-hq/synapse#17447)) - Add Account Data extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17477](element-hq/synapse#17477)) - Add receipts extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17489](element-hq/synapse#17489)) - Add typing notification extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17505](element-hq/synapse#17505)) ### Bugfixes - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](element-hq/synapse#17450)) - Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](element-hq/synapse#17499)) ### Improved Documentation - Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](element-hq/synapse#17476)) ### Internal Changes - Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](element-hq/synapse#17452)) - Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](element-hq/synapse#17478)) - Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](element-hq/synapse#17479)) - Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](element-hq/synapse#17481), [\#17482](element-hq/synapse#17482)) - Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](element-hq/synapse#17501)) - Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](element-hq/synapse#17504)) - Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](element-hq/synapse#17507)) - Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](element-hq/synapse#17511)) - Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](element-hq/synapse#17529)) ### Updates to locked dependencies * Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](element-hq/synapse#17495)) * Bump black from 24.4.2 to 24.8.0. ([\#17522](element-hq/synapse#17522)) * Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](element-hq/synapse#17521)) * Bump ruff from 0.5.4 to 0.5.5. ([\#17494](element-hq/synapse#17494)) * Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](element-hq/synapse#17493)) * Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](element-hq/synapse#17525)) * Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](element-hq/synapse#17523)) * Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](element-hq/synapse#17496)) * Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](element-hq/synapse#17497))

erikjohnston added 8 commits August 14, 2024 18:44

Migrate to per-connection state class

da5339d

Record with new class

baac6c5

Revamp

0561c86

WIP receipts reading

c15b8b3

WIP comments

a1b75f7

Record state

6b9d244

Newsfile

2e7672d

Add tests

55feaae

erikjohnston added the A-Sync label Aug 15, 2024

erikjohnston added 3 commits August 15, 2024 12:42

Newsfile

614c0d7

Comments

100927d

Add proper DB function for getting receipts between things

70d32fb

erikjohnston marked this pull request as ready for review August 15, 2024 14:21

erikjohnston requested a review from a team as a code owner August 15, 2024 14:21

erikjohnston requested a review from MadLittleMods August 15, 2024 14:31

MadLittleMods requested changes Aug 16, 2024

View reviewed changes

erikjohnston and others added 8 commits August 16, 2024 10:07

Update synapse/handlers/sliding_sync.py

64310ec

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

Update synapse/handlers/sliding_sync.py

79e80eb

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

Update synapse/handlers/sliding_sync.py

d982efe

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

Apply suggestions from code review

b0a5c0e

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

Simplify get room status

577370a

Add docstring

27b7a4a

Fixups

dec5314

Rename

5b6755a

erikjohnston mentioned this pull request Aug 16, 2024

Sliding Sync: Handle timeline limit changes (take 2) #17579

Merged

erikjohnston added 5 commits August 16, 2024 15:11

Merge remote-tracking branch 'origin/erikj/ss_per_state_class' into e…

9241da0

…rikj/ss_receipts2

Wrap comment

8a8df06

Docstrings

db4c733

Update names for per-conn state variables

39bece9

Add back in receipts comments

161275d

erikjohnston and others added 4 commits August 16, 2024 15:28

Actually only check for missing receipts in rooms with LIVE

f511576

Apply suggestions from code review

90ecd9e

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

Rewrap comments

49c2f73

Add comment about checking room.initial flag

53a35e8

erikjohnston requested a review from MadLittleMods August 16, 2024 14:45

Add some comments that chain map won't infinitely grow

7f5bccc

MadLittleMods reviewed Aug 19, 2024

View reviewed changes

MadLittleMods approved these changes Aug 19, 2024

View reviewed changes

erikjohnston and others added 3 commits August 19, 2024 20:06

Apply suggestions from code review

f628b8d

Co-authored-by: Eric Eastwood <eric.eastwood@beta.gouv.fr>

Rename new_connection_state

41d37b3

Add receipts to docstring

96aaed8

Base automatically changed from erikj/ss_per_state_class to develop August 19, 2024 19:09

erikjohnston added 2 commits August 19, 2024 20:14

Merge branch 'erikj/ss_per_state_class' into erikj/ss_receipts2

1937ac9

Merge remote-tracking branch 'origin/develop' into erikj/ss_receipts2

4f08d04

erikjohnston merged commit 8b8d74d into develop Aug 19, 2024
39 checks passed

erikjohnston deleted the erikj/ss_receipts2 branch August 19, 2024 20:16

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sliding sync: Correctly track which read receipts we have or have not sent down. #17575

Sliding sync: Correctly track which read receipts we have or have not sent down. #17575

erikjohnston commented Aug 15, 2024 •

edited by MadLittleMods

Loading

MadLittleMods Aug 15, 2024

MadLittleMods Aug 19, 2024

MadLittleMods Aug 15, 2024

MadLittleMods Aug 16, 2024

erikjohnston Aug 16, 2024

MadLittleMods Aug 19, 2024

		@@ -3128,23 +3221,28 @@ class PerConnectionState:
		rooms: The status of each room for the events stream.

Sliding sync: Correctly track which read receipts we have or have not sent down. #17575

Sliding sync: Correctly track which read receipts we have or have not sent down. #17575

Conversation

erikjohnston commented Aug 15, 2024 • edited by MadLittleMods Loading

MadLittleMods Aug 15, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 19, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 15, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 16, 2024

Choose a reason for hiding this comment

erikjohnston Aug 16, 2024

Choose a reason for hiding this comment

MadLittleMods Aug 19, 2024

Choose a reason for hiding this comment

erikjohnston commented Aug 15, 2024 •

edited by MadLittleMods

Loading