Sliding Sync: Update filters to be robust against remote invite rooms #17450

MadLittleMods · 2024-07-16T05:34:58Z

Update filters.is_encrypted and filters.types/filters.not_types to be robust when dealing with remote invite rooms in Sliding Sync.

Part of MSC3575: Sliding Sync

Follow-up to #17434

We now take into account current state, fallback to stripped state for invite/knock rooms, then historical state. If we can't determine the info needed to filter a room (either from state or stripped state), it is filtered out.

Dev notes

Sentinel values:

"mypy sentinel narrowing"
Narrow x is Sentinel where x is Union[type[Sentinel], ...] and Sentinel is @final python/mypy#15553

SQL LATERAL joins:

Example LATERAL join used in this PR -> is_local_host_in_rooms(...)
https://sqlfordevs.com/for-each-loop-lateral-join
source code example in Dendrite
source code example in the Sliding Sync proxy

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
- Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry.
Code style is correct
(run the linters)

synapse/storage/databases/main/state.py

MadLittleMods · 2024-07-16T22:04:00Z

synapse/handlers/sliding_sync.py

+class Sentinel(enum.Enum):
+    # defining a sentinel in this way allows mypy to correctly handle the
+    # type of a dictionary lookup and subsequent type narrowing.
+    UNSET_SENTINEL = object()


This currently works and aligns with other sentinels we already have in the codebase. But I can't use the same pattern in synapse/storage/databases/main/state.py because we need an immutable value to cache.

I can't figure out a way that has these mypy type narrowing benefits while also being immutable.

Seems like we need python/mypy#15553 or adjust our custom mypy plugin that checks whether something is_cacheable (not sure exactly what to adjust, maybe a new custom Synapse Sentinel type that we add to IMMUTABLE_CUSTOM_TYPES).

In any case, it works and we could just merge but it would be nice to find a pattern to use everywhere.

synapse/storage/_base.py

synapse/storage/databases/main/cache.py

synapse/storage/databases/main/state.py

synapse/handlers/sliding_sync.py

erikjohnston · 2024-07-18T10:40:23Z

synapse/handlers/sliding_sync.py

+                    f"Stripped state left unset for room {room_id}. "
+                    + "Make sure you're calling `_bulk_get_stripped_state_for_rooms(...)` "
+                    + "with that room_id. (this is a problem with Synapse itself)"
+                )


Is this true that either the server is in the room or the membership is INVITE/KNOCK? I'm wondering about the case of e.g. leaves where the server is no longer in the room?

This code does work fine when the server leaves the room (added some tests for that). For the non-invite/knock rooms, room_id_to_stripped_state_map[room_id] will be set to None and won't hit this UNSET_SENTINEL.

That does bring up a good point though. Ideally, we shouldn't be using current state for LEAVE/BAN rooms.

I think we can assume the room type isn't secret since it's defined at the the start of the room in the m.room.create event meaning if you've had any membership before, you already knew about it and for invite/knock it is handed out as one of the stripped events anyway.

For the encrypted state, most of the time it's set right away when the room is created but it's possible to set it later. Can we assume it's not considered leaking data? I would say yes especially since it's one of the stripped state events that's just given away.

If we can make these assumptions, we can keep our bulk lookup shortcuts which would be nice.

erikjohnston · 2024-07-18T10:46:24Z

Also, wonder if its worth worrying too much about the encrypted filter? Not sure anyone actually uses it?

See #17450 (comment)

…_map`

MadLittleMods · 2024-07-23T02:00:12Z

synapse/storage/databases/main/state.py

-            room_type = create_event.content.get(EventContentFields.ROOM_TYPE)
-            results[room_id] = room_type
+            return results
+


Should we have a shortcut to check the get_partial_current_state_ids cache? Worth it?

Conflicts: synapse/handlers/sliding_sync.py

See #17450 (comment) This allows us to get rid of the type ignore comments and standardize

synapse/handlers/sliding_sync.py

MadLittleMods · 2024-07-24T16:23:49Z

synapse/handlers/sliding_sync.py

+        # Update our `room_id_to_content` map based on the state at the time of
+        # the membership event.
+        for room_id in rooms_ids_without_stripped_state:
+            # TODO: It would be nice to look this up in a bulk way (N+1 queries)


In the future, we do need to figure out how to bulk fetch historical room state. Looking at historical state is currently a N+1 last ditch effort so we hope it won't happen often but I think it might happen more in practice for the large accounts. For left rooms, only newly_left rooms where the server has left the room (because no one local is still joined) will touch here but that will will only affect a single sync request and will actually be few and far between (this is fine). I think the real problem here will be ~~invited~~/banned rooms that the server has left because these will show up in sync every time (given the range) and hit the historical state path every single time.

Fetching historical state via get_state_at(...) currently involves, getting the latest event at or below the given stream position, looking up state groups for that event, looking up the event_ids in that state group, and then the final full events. And this doesn't even give us a correct "current" state answer (in DAG branching scenarios) but we call it good enough.

I'm wondering if we should just ignore this situation? For remote invites this won't work anyway, as we won't have the state. There's also a slight open question which is if we aren't given a stripped state then should we actually (ab)use any existing knowledge of the room here?

I'm somewhat biasing to no do this due to the performance penalty here.

Is there a way to tell if it's a remote invite?

~~We don't set stripped state for local invites so it's hard to rely just on Membership.INVITE~~

At this point we've already checked the current state and stripped state, so I guess the edge case here is if the user received a local invite but the server has since left the room? TBH I'm not sure that invite would even work anymore. I don't think we care that much about this edge case?

From discussions: the problem is if you leave a room and you're the last person on the server then we'll need to get the state.

@erikjohnston and I discussed this a little further:

To make this fast, we could have new database table that has relevant filter/sort info pre-populated. We would want two tables:

sliding_sync_joined_rooms: A table for storing room meta data that the local server is still participating in. The info here can be shared across all Membership.JOIN. Keyed on (room_id) and updated when the relevant room current state changes

sliding_sync_non_join_memberships: A table for storing meta data for non-join memberships. Keyed on (room_id, user_id) or maybe (room_id, membership_event_id) and set once when the membership occurs.

We need to store non-joins separately because they each need to point to the historical state at the time and we will lose the current_state_events if no one local is joined to the room. A server is only considered participating in a room if someone local is joined.

Separately, @erikjohnston had a very clever observation that we only ever need to decide whether to filter a room or not if we have an update for it and it has NEVER been sent down the connection before (if its anything else we know we can include it).

This has the assumption that the filters don't change between requests. So we either have to track which filters people use per-connection or just say, don't change them otherwise dragons. We can of course can opt for the easy option and add in the filter tracking in the future.

New pre-populated database tables added in #17512

Conflicts: synapse/handlers/sliding_sync.py

See #17450 (comment)

synapse/handlers/sliding_sync.py

erikjohnston · 2024-07-30T13:41:50Z

synapse/handlers/sliding_sync.py

+        # Update our `room_id_to_content` map based on the state at the time of
+        # the membership event.
+        for room_id in rooms_ids_without_stripped_state:
+            # TODO: It would be nice to look this up in a bulk way (N+1 queries)


I'm wondering if we should just ignore this situation? For remote invites this won't work anyway, as we won't have the state. There's also a slight open question which is if we aren't given a stripped state then should we actually (ab)use any existing knowledge of the room here?

I'm somewhat biasing to no do this due to the performance penalty here.

Co-authored-by: Erik Johnston <erikj@element.io>

Conflicts: synapse/handlers/sliding_sync.py

MadLittleMods · 2024-07-30T18:21:00Z

Thanks for the review @erikjohnston 🐫

No significant changes since 1.113.0rc1. - Track which rooms have been sent to clients in the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17447](element-hq/synapse#17447)) - Add Account Data extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17477](element-hq/synapse#17477)) - Add receipts extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17489](element-hq/synapse#17489)) - Add typing notification extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17505](element-hq/synapse#17505)) - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](element-hq/synapse#17450)) - Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](element-hq/synapse#17499)) - Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](element-hq/synapse#17476)) - Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](element-hq/synapse#17452)) - Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](element-hq/synapse#17478)) - Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](element-hq/synapse#17479)) - Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](element-hq/synapse#17481), [\#17482](element-hq/synapse#17482)) - Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](element-hq/synapse#17501)) - Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](element-hq/synapse#17504)) - Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](element-hq/synapse#17507)) - Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](element-hq/synapse#17511)) - Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](element-hq/synapse#17529)) * Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](element-hq/synapse#17495)) * Bump black from 24.4.2 to 24.8.0. ([\#17522](element-hq/synapse#17522)) * Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](element-hq/synapse#17521)) * Bump ruff from 0.5.4 to 0.5.5. ([\#17494](element-hq/synapse#17494)) * Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](element-hq/synapse#17493)) * Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](element-hq/synapse#17525)) * Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](element-hq/synapse#17523)) * Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](element-hq/synapse#17496)) * Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](element-hq/synapse#17497))

) Pre-populate room data for quick filtering/sorting in the Sliding Sync API Spawning from #17450 (comment) This PR is acting as the Synapse version `N+1` step in the gradual migration being tracked by #17623 Adding two new database tables: - `sliding_sync_joined_rooms`: A table for storing room meta data that the local server is still participating in. The info here can be shared across all `Membership.JOIN`. Keyed on `(room_id)` and updated when the relevant room current state changes or a new event is sent in the room. - `sliding_sync_membership_snapshots`: A table for storing a snapshot of room meta data at the time of the local user's membership. Keyed on `(room_id, user_id)` and only updated when a user's membership in a room changes. Also adds background updates to populate these tables with all of the existing data. We want to have the guarantee that if a row exists in the sliding sync tables, we are able to rely on it (accurate data). And if a row doesn't exist, we use a fallback to get the same info until the background updates fill in the rows or a new event comes in triggering it to be fully inserted. This means we need a couple extra things in place until we bump `SCHEMA_COMPAT_VERSION` and run the foreground update in the `N+2` part of the gradual migration. For context on why we can't rely on the tables without these things see [1]. 1. On start-up, block until we clear out any rows for the rooms that have had events since the max-`stream_ordering` of the `sliding_sync_joined_rooms` table (compare to max-`stream_ordering` of the `events` table). For `sliding_sync_membership_snapshots`, we can compare to the max-`stream_ordering` of `local_current_membership` - This accounts for when someone downgrades their Synapse version and then upgrades it again. This will ensure that we don't have any stale/out-of-date data in the `sliding_sync_joined_rooms`/`sliding_sync_membership_snapshots` tables since any new events sent in rooms would have also needed to be written to the sliding sync tables. For example a new event needs to bump `event_stream_ordering` in `sliding_sync_joined_rooms` table or some state in the room changing (like the room name). Or another example of someone's membership changing in a room affecting `sliding_sync_membership_snapshots`. 1. Add another background update that will catch-up with any rows that were just deleted from the sliding sync tables (based on the activity in the `events`/`local_current_membership`). The rooms that need recalculating are added to the `sliding_sync_joined_rooms_to_recalculate` table. 1. Making sure rows are fully inserted. Instead of partially inserting, we need to check if the row already exists and fully insert all data if not. All of this extra functionality can be removed once the `SCHEMA_COMPAT_VERSION` is bumped with support for the new sliding sync tables so people can no longer downgrade (the `N+2` part of the gradual migration). <details> <summary><sup>[1]</sup></summary> For `sliding_sync_joined_rooms`, since we partially insert rows as state comes in, we can't rely on the existence of the row for a given `room_id`. We can't even rely on looking at whether the background update has finished. There could still be partial rows from when someone reverted their Synapse version after the background update finished, had some state changes (or new rooms), then upgraded again and more state changes happen leaving a partial row. For `sliding_sync_membership_snapshots`, we insert items as a whole except for the `forgotten` column ~~so we can rely on rows existing and just need to always use a fallback for the `forgotten` data. We can't use the `forgotten` column in the table for the same reasons above about `sliding_sync_joined_rooms`.~~ We could have an out-of-date membership from when someone reverted their Synapse version. (same problems as outlined for `sliding_sync_joined_rooms` above) Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7) </details> ### TODO - [x] Update `stream_ordering`/`bump_stamp` - [x] Handle remote invites - [x] Handle state resets - [x] Consider adding `sender` so we can filter `LEAVE` memberships and distinguish from kicks. - [x] We should add it to be able to tell leaves from kicks - [x] Consider adding `tombstone` state to help address #17540 - [x] We should add it `tombstone_successor_room_id` - [x] Consider adding `forgotten` status to avoid extra lookup/table-join on `room_memberships` - [x] We should add it - [x] Background update to fill in values for all joined rooms and non-join membership - [x] Clean-up tables when room is deleted - [ ] Make sure tables are useful to our use case - First explored in https://github.com/element-hq/synapse/compare/erikj/ss_use_new_tables - Also explored in 76b5a57 - [x] Plan for how can we use this with a fallback - See plan discussed above in main area of the issue description - Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.dz5x6ef4mxz7) - [x] Plan for how we can rely on this new table without a fallback - Synapse version `N+1`: (this PR) Bump `SCHEMA_VERSION` to `87`. Add new tables and background update to backfill all rows. Since this is a new table, we don't have to add any `NOT VALID` constraints and validate them when the background update completes. Read from new tables with a fallback in cases where the rows aren't filled in yet. - Synapse version `N+2`: Bump `SCHEMA_VERSION` to `88` and bump `SCHEMA_COMPAT_VERSION` to `87` because we don't want people to downgrade and miss writes while they are on an older version. Add a foreground update to finish off the backfill so we can read from new tables without the fallback. Application code can now rely on the new tables being populated. - Discussed in an [internal meeting](https://docs.google.com/document/d/1MnuvPkaCkT_wviSQZ6YKBjiWciCBFMd-7hxyCO-OCbQ/edit#bookmark=id.hh7shg4cxdhj) ### Dev notes ``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase SYNAPSE_POSTGRES=1 SYNAPSE_POSTGRES_USER=postgres SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.storage.test_events.SlidingSyncPrePopulatedTablesTestCase ``` ``` SYNAPSE_TEST_LOG_LEVEL=INFO poetry run trial tests.handlers.test_sliding_sync.FilterRoomsTestCase ``` Reference: - [Development docs on background updates and worked examples of gradual migrations ](https://github.com/element-hq/synapse/blob/1dfa59b238cee0dc62163588cc9481896c288979/docs/development/database_schema.md#background-updates) - A real example of a gradual migration: matrix-org/synapse#15649 (comment) - Adding `rooms.creator` field that needed a background update to backfill data, matrix-org/synapse#10697 - Adding `rooms.room_version` that needed a background update to backfill data, matrix-org/synapse#6729 - Adding `room_stats_state.room_type` that needed a background update to backfill data, matrix-org/synapse#13031 - Tables from MSC2716: `insertion_events`, `insertion_event_edges`, `insertion_event_extremities`, `batch_events` - `current_state_events` updated in `synapse/storage/databases/main/events.py` --- ``` persist_event (adds to queue) _persist_event_batch _persist_events_and_state_updates (assigns `stream_ordering` to events) _persist_events_txn _store_event_txn _update_metadata_tables_txn _store_room_members_txn _update_current_state_txn ``` --- > Concatenated Indexes [...] (also known as multi-column, composite or combined index) > > [...] key consists of multiple columns. > > We can take advantage of the fact that the first index column is always usable for searching > > *-- https://use-the-index-luke.com/sql/where-clause/the-equals-operator/concatenated-keys* --- Dealing with `portdb` (`synapse/_scripts/synapse_port_db.py`), #17512 (comment) --- <details> <summary>SQL queries:</summary> Both of these are equivalent and work in SQLite and Postgres Options 1: ```sql WITH data_table (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) AS ( VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ) INSERT INTO sliding_sync_non_join_memberships (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) SELECT * FROM data_table WHERE membership != ? ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` Option 2: ```sql INSERT INTO sliding_sync_non_join_memberships (room_id, user_id, membership_event_id, membership, event_stream_ordering, {", ".join(insert_keys)}) SELECT column1 as room_id, column2 as user_id, column3 as membership_event_id, column4 as membership, column5 as event_stream_ordering, {", ".join("column" + str(i) for i in range(6, 6 + len(insert_keys)))} FROM ( VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ) as v WHERE membership != ? ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` If we don't need the `membership` condition, we could use: ```sql INSERT INTO sliding_sync_non_join_memberships (room_id, membership_event_id, user_id, membership, event_stream_ordering, {", ".join(insert_keys)}) VALUES ( ?, ?, ?, (SELECT membership FROM room_memberships WHERE event_id = ?), (SELECT stream_ordering FROM events WHERE event_id = ?), {", ".join("?" for _ in insert_values)} ) ON CONFLICT (room_id, user_id) DO UPDATE SET membership_event_id = EXCLUDED.membership_event_id, membership = EXCLUDED.membership, event_stream_ordering = EXCLUDED.event_stream_ordering, {", ".join(f"{key} = EXCLUDED.{key}" for key in insert_keys)} ``` </details> ### Pull Request Checklist  * [x] Pull request is based on the develop branch * [x] Pull request includes a [changelog file](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#changelog). The entry should: - Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from `EventStore` to `EventWorkerStore`.". - Use markdown where necessary, mostly for `code blocks`. - End with either a period (.) or an exclamation mark (!). - Start with a capital letter. - Feel free to credit yourself, by adding a sentence "Contributed by @github_username." or "Contributed by [Your Name]." to the end of the entry. * [x] [Code style](https://element-hq.github.io/synapse/latest/code_style.html) is correct (run the [linters](https://element-hq.github.io/synapse/latest/development/contributing_guide.html#run-the-linters)) --------- Co-authored-by: Erik Johnston <erik@matrix.org>

@kfiven

# Synapse 1.114.0 (2024-09-02) This release enables support for [MSC4186](matrix-org/matrix-spec-proposals#4186) — Simplified Sliding Sync. This allows using the upcoming releases of the Element X mobile apps without having to run a Sliding Sync Proxy. ### Features - Enable native sliding sync support ([MSC3575](matrix-org/matrix-spec-proposals#3575) and [MSC4186](matrix-org/matrix-spec-proposals#4186)) by default. ([\#17648](element-hq/synapse#17648)) # Synapse 1.114.0rc3 (2024-08-30) ### Bugfixes - Fix regression in v1.114.0rc2 that caused workers to fail to start. ([\#17626](element-hq/synapse#17626)) # Synapse 1.114.0rc2 (2024-08-30) ### Features - Improve cross-signing upload when using [MSC3861](matrix-org/matrix-spec-proposals#3861) to use a custom UIA flow stage, with web fallback support. ([\#17509](element-hq/synapse#17509)) - Make `hash_password` script accept password input from stdin. ([\#17608](element-hq/synapse#17608)) ### Bugfixes - Fix hierarchy returning 403 when room is accessible through federation. Contributed by Krishan (@kfiven). ([\#17194](element-hq/synapse#17194)) - Fix content-length on federation `/thumbnail` responses. ([\#17532](element-hq/synapse#17532)) - Fix authenticated media responses using a wrong limit when following redirects over federation. ([\#17543](element-hq/synapse#17543)) ### Internal Changes - MSC3861: load the issuer and account management URLs from OIDC discovery. ([\#17407](element-hq/synapse#17407)) - Refactor sliding sync class into multiple files. ([\#17595](element-hq/synapse#17595)) - Store sliding sync per-connection state in the database. ([\#17599](element-hq/synapse#17599)) - Make the sliding sync `PerConnectionState` class immutable. ([\#17600](element-hq/synapse#17600)) - Add support to `@tag_args` for standalone functions. ([\#17604](element-hq/synapse#17604)) - Speed up incremental syncs in sliding sync by adding some more caching. ([\#17606](element-hq/synapse#17606)) - Always return the user's own read receipts in sliding sync. ([\#17617](element-hq/synapse#17617)) - Replace `isort` and `black` with `ruff`. ([\#17620](element-hq/synapse#17620)) - Refactor sliding sync code to move room list logic out into a separate class. ([\#17622](element-hq/synapse#17622)) ### Updates to locked dependencies * Bump attrs from 23.2.0 to 24.2.0. ([\#17609](element-hq/synapse#17609)) * Bump cryptography from 42.0.8 to 43.0.0. ([\#17584](element-hq/synapse#17584)) * Bump phonenumbers from 8.13.43 to 8.13.44. ([\#17610](element-hq/synapse#17610)) * Bump pygithub from 2.3.0 to 2.4.0. ([\#17612](element-hq/synapse#17612)) * Bump pyyaml from 6.0.1 to 6.0.2. ([\#17611](element-hq/synapse#17611)) * Bump sentry-sdk from 2.12.0 to 2.13.0. ([\#17585](element-hq/synapse#17585)) * Bump serde from 1.0.206 to 1.0.208. ([\#17581](element-hq/synapse#17581)) * Bump serde from 1.0.208 to 1.0.209. ([\#17613](element-hq/synapse#17613)) * Bump serde_json from 1.0.124 to 1.0.125. ([\#17582](element-hq/synapse#17582)) * Bump serde_json from 1.0.125 to 1.0.127. ([\#17614](element-hq/synapse#17614)) * Bump types-jsonschema from 4.23.0.20240712 to 4.23.0.20240813. ([\#17583](element-hq/synapse#17583)) * Bump types-setuptools from 71.1.0.20240726 to 71.1.0.20240818. ([\#17586](element-hq/synapse#17586)) # Synapse 1.114.0rc1 (2024-08-20) ### Features - Add a flag to `/versions`, `org.matrix.simplified_msc3575`, to indicate whether experimental sliding sync support has been enabled. ([\#17571](element-hq/synapse#17571)) - Handle changes in `timeline_limit` in experimental sliding sync. ([\#17579](element-hq/synapse#17579)) - Correctly track read receipts that should be sent down in experimental sliding sync. ([\#17575](element-hq/synapse#17575), [\#17589](element-hq/synapse#17589), [\#17592](element-hq/synapse#17592)) ### Bugfixes - Start handlers for new media endpoints when media resource configured. ([\#17483](element-hq/synapse#17483)) - Fix timeline ordering (using `stream_ordering` instead of topological ordering) in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17510](element-hq/synapse#17510)) - Fix experimental sliding sync implementation to remember any updates in rooms that were not sent down immediately. ([\#17535](element-hq/synapse#17535)) - Better exclude partially stated rooms if we must await full state in experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17538](element-hq/synapse#17538)) - Handle lower-case http headers in `_Mulitpart_Parser_Protocol`. ([\#17545](element-hq/synapse#17545)) - Fix fetching federation signing keys from servers that omit `old_verify_keys`. Contributed by @tulir @ Beeper. ([\#17568](element-hq/synapse#17568)) - Fix bug where we would respond with an error when a remote server asked for media that had a length of 0, using the new multipart federation media endpoint. ([\#17570](element-hq/synapse#17570)) ### Improved Documentation - Clarify default behaviour of the [`auto_accept_invites.worker_to_run_on`](https://element-hq.github.io/synapse/develop/usage/configuration/config_documentation.html#auto-accept-invites) option. ([\#17515](element-hq/synapse#17515)) - Improve docstrings for profile methods. ([\#17559](element-hq/synapse#17559)) ### Internal Changes - Add more tracing to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17514](element-hq/synapse#17514)) - Fixup comment in sliding sync implementation. ([\#17531](element-hq/synapse#17531)) - Replace override of deprecated method `HTTPAdapter.get_connection` with `get_connection_with_tls_context`. ([\#17536](element-hq/synapse#17536)) - Fix performance of device lists in `/key/changes` and sliding sync. ([\#17537](element-hq/synapse#17537), [\#17548](element-hq/synapse#17548)) - Bump setuptools from 67.6.0 to 72.1.0. ([\#17542](element-hq/synapse#17542)) - Add a utility function for generating random event IDs. ([\#17557](element-hq/synapse#17557)) - Speed up responding to media requests. ([\#17558](element-hq/synapse#17558), [\#17561](element-hq/synapse#17561), [\#17564](element-hq/synapse#17564), [\#17566](element-hq/synapse#17566), [\#17567](element-hq/synapse#17567), [\#17569](element-hq/synapse#17569)) - Test github token before running release script steps. ([\#17562](element-hq/synapse#17562)) - Reduce log spam of multipart files. ([\#17563](element-hq/synapse#17563)) - Refactor per-connection state in experimental sliding sync handler. ([\#17574](element-hq/synapse#17574)) - Add histogram metrics for sliding sync processing time. ([\#17593](element-hq/synapse#17593)) ### Updates to locked dependencies * Bump bytes from 1.6.1 to 1.7.1. ([\#17526](element-hq/synapse#17526)) * Bump lxml from 5.2.2 to 5.3.0. ([\#17550](element-hq/synapse#17550)) * Bump phonenumbers from 8.13.42 to 8.13.43. ([\#17551](element-hq/synapse#17551)) * Bump regex from 1.10.5 to 1.10.6. ([\#17527](element-hq/synapse#17527)) * Bump sentry-sdk from 2.10.0 to 2.12.0. ([\#17553](element-hq/synapse#17553)) * Bump serde from 1.0.204 to 1.0.206. ([\#17556](element-hq/synapse#17556)) * Bump serde_json from 1.0.122 to 1.0.124. ([\#17555](element-hq/synapse#17555)) * Bump sigstore/cosign-installer from 3.5.0 to 3.6.0. ([\#17549](element-hq/synapse#17549)) * Bump types-pyyaml from 6.0.12.20240311 to 6.0.12.20240808. ([\#17552](element-hq/synapse#17552)) * Bump types-requests from 2.31.0.20240406 to 2.32.0.20240712. ([\#17524](element-hq/synapse#17524)) # Synapse 1.113.0 (2024-08-13) No significant changes since 1.113.0rc1. # Synapse 1.113.0rc1 (2024-08-06) ### Features - Track which rooms have been sent to clients in the experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17447](element-hq/synapse#17447)) - Add Account Data extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17477](element-hq/synapse#17477)) - Add receipts extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17489](element-hq/synapse#17489)) - Add typing notification extension support to experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint. ([\#17505](element-hq/synapse#17505)) ### Bugfixes - Update experimental [MSC3575](matrix-org/matrix-spec-proposals#3575) Sliding Sync `/sync` endpoint to handle invite/knock rooms when filtering. ([\#17450](element-hq/synapse#17450)) - Fix a bug introduced in v1.110.0 which caused `/keys/query` to return incomplete results, leading to high network activity and CPU usage on Matrix clients. ([\#17499](element-hq/synapse#17499)) ### Improved Documentation - Update the [`allowed_local_3pids`](https://element-hq.github.io/synapse/v1.112/usage/configuration/config_documentation.html#allowed_local_3pids) config option's msisdn address to a working example. ([\#17476](element-hq/synapse#17476)) ### Internal Changes - Change sliding sync to use their own token format in preparation for storing per-connection state. ([\#17452](element-hq/synapse#17452)) - Ensure we don't send down negative `bump_stamp` in experimental sliding sync endpoint. ([\#17478](element-hq/synapse#17478)) - Do not send down empty room entries down experimental sliding sync endpoint. ([\#17479](element-hq/synapse#17479)) - Refactor Sliding Sync tests to better utilize the `SlidingSyncBase`. ([\#17481](element-hq/synapse#17481), [\#17482](element-hq/synapse#17482)) - Add some opentracing tags and logging to the experimental sliding sync implementation. ([\#17501](element-hq/synapse#17501)) - Split and move Sliding Sync tests so we have some more sane test file sizes. ([\#17504](element-hq/synapse#17504)) - Update the `limited` field description in the Sliding Sync response to accurately describe what it actually represents. ([\#17507](element-hq/synapse#17507)) - Easier to understand `timeline` assertions in Sliding Sync tests. ([\#17511](element-hq/synapse#17511)) - Reset the sliding sync connection if we don't recognize the per-connection state position. ([\#17529](element-hq/synapse#17529)) ### Updates to locked dependencies * Bump bcrypt from 4.1.3 to 4.2.0. ([\#17495](element-hq/synapse#17495)) * Bump black from 24.4.2 to 24.8.0. ([\#17522](element-hq/synapse#17522)) * Bump phonenumbers from 8.13.39 to 8.13.42. ([\#17521](element-hq/synapse#17521)) * Bump ruff from 0.5.4 to 0.5.5. ([\#17494](element-hq/synapse#17494)) * Bump serde_json from 1.0.120 to 1.0.121. ([\#17493](element-hq/synapse#17493)) * Bump serde_json from 1.0.121 to 1.0.122. ([\#17525](element-hq/synapse#17525)) * Bump towncrier from 23.11.0 to 24.7.1. ([\#17523](element-hq/synapse#17523)) * Bump types-pyopenssl from 24.1.0.20240425 to 24.1.0.20240722. ([\#17496](element-hq/synapse#17496)) * Bump types-setuptools from 70.1.0.20240627 to 71.1.0.20240726. ([\#17497](element-hq/synapse#17497))

MadLittleMods added 8 commits July 15, 2024 15:13

Make filters.is_encrypted robust to remote invites

3a6eec1

Try share work

7f72d80

More robust

18ec437

Fix lints

a67157e

Return bool from db

ffb61cf

Add tests

9b77c97

Standardize

d609205

Add TODO for more tests

8ca8ae5

MadLittleMods added the A-Sync label Jul 16, 2024

Add changelog

4f50054

MadLittleMods commented Jul 16, 2024

View reviewed changes

synapse/storage/databases/main/state.py Outdated Show resolved Hide resolved

MadLittleMods mentioned this pull request Jul 16, 2024

Fix filtering room types on remote rooms #17434

Merged

MadLittleMods added 4 commits July 16, 2024 14:56

Invalidate caches

7ae2425

Use ROOM_UNKNOWN_SENTINEL

b35a2da

Iterate on sentinels

4bb34d3

Simply sentinel for now

a07ee22

MadLittleMods commented Jul 16, 2024

View reviewed changes

MadLittleMods added 3 commits July 16, 2024 17:04

Merge branch 'develop' into madlittlemods/robust-remote-invite-filters

2c9ec61

fetch -> get

27a97a0

Update comments

517bfc4

MadLittleMods commented Jul 16, 2024

View reviewed changes

synapse/storage/_base.py Show resolved Hide resolved

MadLittleMods commented Jul 16, 2024

View reviewed changes

synapse/storage/databases/main/cache.py Outdated Show resolved Hide resolved

MadLittleMods added 3 commits July 16, 2024 17:15

Fix typo

d22bae4

Better docstring

8624fb0

Use constants

a77b70e

MadLittleMods marked this pull request as ready for review July 17, 2024 00:06

MadLittleMods requested a review from a team as a code owner July 17, 2024 00:06

erikjohnston reviewed Jul 18, 2024

View reviewed changes

Prefer not implementing if not used

df5093b

See #17450 (comment)

MadLittleMods added 8 commits July 22, 2024 19:19

Add `_bulk_get_partial_current_state_content_for_rooms_from_sync_room…

05bbcb0

…_map`

Simplify and share

376d900

Remove unused methods

28a36e8

Maybe current state in the future

ac7ca67

Fix cache busting

15e157e

Fix lints

19968e1

Shorten name

8f5e4bf

Reconsider all stripped events

3c144b3

MadLittleMods commented Jul 23, 2024

View reviewed changes

Merge branch 'develop' into madlittlemods/robust-remote-invite-filters

f3b4c9f

Conflicts: synapse/handlers/sliding_sync.py

MadLittleMods requested a review from erikjohnston July 23, 2024 02:06

Prefer cache invalidation helper

1732b02

See #17450 (comment) This allows us to get rid of the type ignore comments and standardize

erikjohnston reviewed Jul 24, 2024

View reviewed changes

synapse/handlers/sliding_sync.py Outdated Show resolved Hide resolved

MadLittleMods commented Jul 24, 2024

View reviewed changes

MadLittleMods added 3 commits July 25, 2024 17:00

Merge branch 'develop' into madlittlemods/robust-remote-invite-filters

20902e6

Conflicts: synapse/handlers/sliding_sync.py

Go back to dedicated room_stats_state lookups

e9b76cc

See #17450 (comment)

Update cache busting

484007e

MadLittleMods requested a review from erikjohnston July 26, 2024 01:27

erikjohnston reviewed Jul 30, 2024

View reviewed changes

Prefer assert_never

7acb1ad

Co-authored-by: Erik Johnston <erikj@element.io>

erikjohnston approved these changes Jul 30, 2024

View reviewed changes

Merge branch 'develop' into madlittlemods/robust-remote-invite-filters

45ed4ce

Conflicts: synapse/handlers/sliding_sync.py

MadLittleMods merged commit 46de0ee into develop Jul 30, 2024
39 checks passed

MadLittleMods deleted the madlittlemods/robust-remote-invite-filters branch July 30, 2024 18:20

MadLittleMods mentioned this pull request Jul 31, 2024

Sliding Sync: Pre-populate room data for quick filtering/sorting #17512

Merged

17 tasks

MadLittleMods mentioned this pull request Sep 18, 2024

@cachedList can insert None instead of absence of a key #17726

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Sliding Sync: Update filters to be robust against remote invite rooms #17450

Sliding Sync: Update filters to be robust against remote invite rooms #17450

MadLittleMods commented Jul 16, 2024 •

edited

Loading

MadLittleMods Jul 16, 2024

erikjohnston Jul 18, 2024

MadLittleMods Jul 19, 2024 •

edited

Loading

erikjohnston commented Jul 18, 2024

MadLittleMods Jul 23, 2024

MadLittleMods Jul 24, 2024 •

edited

Loading

erikjohnston Jul 30, 2024

MadLittleMods Jul 30, 2024 •

edited

Loading

erikjohnston Jul 30, 2024

erikjohnston Jul 30, 2024

MadLittleMods Jul 30, 2024

MadLittleMods Sep 3, 2024

erikjohnston Jul 30, 2024

MadLittleMods commented Jul 30, 2024

Sliding Sync: Update filters to be robust against remote invite rooms #17450

Sliding Sync: Update filters to be robust against remote invite rooms #17450

Conversation

MadLittleMods commented Jul 16, 2024 • edited Loading

Dev notes

Pull Request Checklist

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Jul 19, 2024 • edited Loading

Choose a reason for hiding this comment

erikjohnston commented Jul 18, 2024

Choose a reason for hiding this comment

MadLittleMods Jul 24, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods Jul 30, 2024 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

MadLittleMods commented Jul 30, 2024

MadLittleMods commented Jul 16, 2024 •

edited

Loading

MadLittleMods Jul 19, 2024 •

edited

Loading

MadLittleMods Jul 24, 2024 •

edited

Loading

MadLittleMods Jul 30, 2024 •

edited

Loading