Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Support MSC3266 room summaries over federation #11507

Merged
merged 19 commits into from
May 5, 2022

Conversation

deepbluev7
Copy link
Contributor

@deepbluev7 deepbluev7 commented Dec 3, 2021

Signed-off-by: Nicolas Werner nicolas.werner@hotmail.de

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
  • Pull request includes a sign off
  • Code style is correct
    (run the linters)

synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
@squahtx squahtx self-assigned this Dec 6, 2021
@squahtx
Copy link
Contributor

squahtx commented Dec 6, 2021

Thank you for getting the last bits of msc3266 implemented!
The PR's on the right track, though I have a few questions.

@deepbluev7
Copy link
Contributor Author

Thank you for the quick review, I'll see if I can get those details fixed over the week!

@anoadragon453 anoadragon453 added the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Jan 10, 2022
deepbluev7 and others added 5 commits April 28, 2022 14:29
@deepbluev7 deepbluev7 force-pushed the nico/room-summary-federation branch from 50746c3 to 7d4bafd Compare May 2, 2022 17:39
@deepbluev7
Copy link
Contributor Author

Since it has been so long, I rebased my changes now and addressed most feedback. It also is adapted to the latest changes on the MSC now. Not sure about the still open review comment though.

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Copy link
Contributor

@squahtx squahtx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the changes. I've forgotten a lot of the context about this PR.

We could do with a test for the new code in tests/handlers/test_room_summary.py.
There are a few examples of mocking federation responses in there.

synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
@squahtx squahtx requested a review from a team May 3, 2022 19:30
@squahtx
Copy link
Contributor

squahtx commented May 3, 2022

@matrix-org/synapse-core Can we get input on #11507 (comment)?

tl;dr. what does is_host_joined return for a remote room that a local user has received an invite to?

deepbluev7 and others added 2 commits May 4, 2022 06:08
Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
@deepbluev7
Copy link
Contributor Author

I added a very basic test based on the room hierarchy tests. It is not very extensive, but most stuff is tested for hierarchies anyway.

@squahtx
Copy link
Contributor

squahtx commented May 4, 2022

Thanks for adding the test!

deepbluev7 added 2 commits May 4, 2022 12:19
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Thanks anoa!

Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
@@ -713,6 +716,8 @@ async def _build_room_entry(self, room_id: str, for_federation: bool) -> JsonDic
),
"guest_can_join": stats["guest_access"] == "can_join",
"room_type": create_event.content.get(EventContentFields.ROOM_TYPE),
"im.nheko.summary.version": stats["version"],
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In some situations this could potentially fix the hack on L656-657 about assuming knock is valid. Would probably not be available often enough though.

We should guard these with the unstable config flag, probably?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think we should wait for this to be stable first and available for a while and then we can use that to fix the hack, since we need the remote server to use it. After a few months it is probably safe to just assume a room is version 1, if no version is returned.

I think an unstable flag would probably make more sense for the whole feature instead of just some namespaced fields. I was hoping the MSC just FCPs and passes in the next few weeks, since it is quite small and then we can just stabilize it (so I don't need to figure out how to add unstable flags :D). But I'll defer that to your judgement. If you want a flag, I can add it. I am just lazy .-.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think an unstable flag would probably make more sense for the whole feature instead of just some namespaced fields. I was hoping the MSC just FCPs and passes in the next few weeks, since it is quite small and then we can just stabilize it (so I don't need to figure out how to add unstable flags :D). But I'll defer that to your judgement. If you want a flag, I can add it. I am just lazy .-.

There already is one for the whole feature -- I'm suggesting to re-use it and guard these fields with it.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Okay, I think I figured it out. I cache the flag in the Handler, as it doesn't cache the hs yet and I think just caching the flag makes more sense? No idea :D

deepbluev7 and others added 5 commits May 4, 2022 15:56
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
Signed-off-by: Nicolas Werner <nicolas.werner@hotmail.de>
@squahtx squahtx requested a review from clokep May 5, 2022 12:09
@squahtx
Copy link
Contributor

squahtx commented May 5, 2022

Otherwise looks good from my end!

Co-authored-by: Sean Quah <8349537+squahtx@users.noreply.github.com>
Copy link
Contributor

@squahtx squahtx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks good from my end!
Awaiting re-review from @clokep before merging.

suggested_only=True,
)

if room_entry:
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm finding the level of indentation here rather difficult to follow. It could be nicer to check if not room_entry and raise immediately, rather than at the bottom of the method, but won't block on that.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I've inverted the logic now and combined the 2 if cases, which gets rid of 2 levels of indentation. If I don't combine them I would have to duplicate the error messages, because I want the error to be the same for inaccessible or not found.

synapse/handlers/room_summary.py Outdated Show resolved Hide resolved
Copy link
Member

@clokep clokep left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks great!

@squahtx
Copy link
Contributor

squahtx commented May 5, 2022

Is this something we want to merge immediately or are we waiting for the MSC process?

@deepbluev7
Copy link
Contributor Author

Half of the implementation is already merged, it is hidden behind a flag and I would prefer to not have to fix conflicts in a few months when (or if) the MSC passes FCP. Imo the MSC also shouldn't change a lot anymore. Some fields might get renamed, but those are namespaced and it shouldn't matter if those then get renamed when stabilizing. It probably doesn't matter much, but from my perspective merging seems simpler :3

@clokep
Copy link
Member

clokep commented May 5, 2022

Is this something we want to merge immediately or are we waiting for the MSC process?

I think it can be merged since it is already partially merged (and behind a config flag).

@clokep clokep removed the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label May 5, 2022
@squahtx squahtx merged commit a377a43 into matrix-org:develop May 5, 2022
DMRobertson pushed a commit that referenced this pull request May 10, 2022
Synapse 1.59.0rc1 (2022-05-10)
==============================

This release makes several changes that server administrators should be aware of:

- Device name lookup over federation is now disabled by default. ([\#12616](#12616))
- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](#12452), [\#12654](#12654))

See [the upgrade notes](https://github.com/matrix-org/synapse/blob/develop/docs/upgrade.md#upgrading-to-v1590) for more details.

Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](#12597))

Features
--------

- Support [MSC3266](matrix-org/matrix-spec-proposals#3266) room summaries over federation. ([\#11507](#11507))
- Implement [changes](matrix-org/matrix-spec-proposals@4a77139) to [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#12168](#12168), [\#12635](#12635), [\#12636](#12636), [\#12670](#12670))
- Extend the [module API](https://github.com/matrix-org/synapse/blob/release-v1.59/synapse/module_api/__init__.py) to allow modules to change actions for existing push rules of local users. ([\#12406](#12406))
- Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. ([\#12452](#12452))
- Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ([\#12654](#12654))
- Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid. ([\#12526](#12526))
- Implement [MSC3786](matrix-org/matrix-spec-proposals#3786): Add a default push rule to ignore `m.room.server_acl` events. ([\#12601](#12601))
- Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice. ([\#12619](#12619))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.48.0 where the latest thread reply provided failed to include the proper bundled aggregations. ([\#12273](#12273))
- Fix a bug introduced in Synapse 1.22.0 where attempting to send a large amount of read receipts to an application service all at once would result in duplicate content and abnormally high memory usage. Contributed by Brad & Nick @ Beeper. ([\#12544](#12544))
- Fix a bug introduced in Synapse 1.57.0 which could cause `Failed to calculate hosts in room` errors to be logged for outbound federation. ([\#12570](#12570))
- Fix a long-standing bug where status codes would almost always get logged as `200!`, irrespective of the actual status code, when clients disconnect before a request has finished processing. ([\#12580](#12580))
- Fix race when persisting an event and deleting a room that could lead to outbound federation breaking. ([\#12594](#12594))
- Fix a bug introduced in Synapse 1.53.0 where bundled aggregations for annotations/edits were incorrectly calculated. ([\#12633](#12633))
- Fix a long-standing bug where rooms containing power levels with string values could not be upgraded. ([\#12657](#12657))
- Prevent memory leak from reoccurring when presence is disabled. ([\#12656](#12656))

Updates to the Docker image
---------------------------

- Explicitly opt-in to using [BuildKit-specific features](https://github.com/moby/buildkit/blob/master/frontend/dockerfile/docs/syntax.md) in the Dockerfile. This fixes issues with building images in some GitLab CI environments. ([\#12541](#12541))
- Update the "Build docker images" GitHub Actions workflow to use `docker/metadata-action` to generate docker image tags, instead of a custom shell script. Contributed by @henryclw. ([\#12573](#12573))

Improved Documentation
----------------------

- Update SQL statements and replace use of old table `user_stats_historical` in docs for Synapse Admins. ([\#12536](#12536))
- Add missing linebreak to `pipx` install instructions. ([\#12579](#12579))
- Add information about the TCP replication module to docs. ([\#12621](#12621))
- Fixes to the formatting of `README.rst`. ([\#12627](#12627))
- Fix docs on how to run specific Complement tests using the `complement.sh` test runner. ([\#12664](#12664))

Deprecations and Removals
-------------------------

- Remove unstable identifiers from [MSC3069](matrix-org/matrix-spec-proposals#3069). ([\#12596](#12596))
- Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from
  [MSC2778](matrix-org/matrix-spec-proposals#2778). ([\#12597](#12597))
- Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk. ([\#12613](#12613))

Internal Changes
----------------

- Use supervisord to supervise Postgres and Caddy in the Complement image to reduce restart time. ([\#12480](#12480))
- Immediately retry any requests that have backed off when a server comes back online. ([\#12500](#12500))
- Use `make_awaitable` instead of `defer.succeed` for return values of mocks in tests. ([\#12505](#12505))
- Consistently check if an object is a `frozendict`. ([\#12564](#12564))
- Protect module callbacks with read semantics against cancellation. ([\#12568](#12568))
- Improve comments and error messages around access tokens. ([\#12577](#12577))
- Improve docstrings for the receipts store. ([\#12581](#12581))
- Use constants for read-receipts in tests. ([\#12582](#12582))
- Log status code of cancelled requests as 499 and avoid logging stack traces for them. ([\#12587](#12587), [\#12663](#12663))
- Remove special-case for `twisted` logger from default log config. ([\#12589](#12589))
- Use `getClientAddress` instead of the deprecated `getClientIP`. ([\#12599](#12599))
- Add link to documentation in Grafana Dashboard. ([\#12602](#12602))
- Reduce log spam when running multiple event persisters. ([\#12610](#12610))
- Add extra debug logging to federation sender. ([\#12614](#12614))
- Prevent remote homeservers from requesting local user device names by default. ([\#12616](#12616))
- Add a consistency check on events which we read from the database. ([\#12620](#12620))
- Remove use of the `constantly` library and switch to enums for `EventRedactBehaviour`. Contributed by @andrewdoh. ([\#12624](#12624))
- Remove unused code related to receipts. ([\#12632](#12632))
- Minor improvements to the scripts for running Synapse in worker mode under Complement. ([\#12637](#12637))
- Move `pympler` back in to the `all` extras. ([\#12652](#12652))
- Fix spelling of `M_UNRECOGNIZED` in comments. ([\#12665](#12665))
- Release script: confirm the commit to be tagged before tagging. ([\#12556](#12556))
- Fix a typo in the announcement text generated by the Synapse release development script. ([\#12612](#12612))

- Fix scripts-dev to pass typechecking. ([\#12356](#12356))
- Add some type hints to datastore. ([\#12485](#12485))
- Remove unused `# type: ignore`s. ([\#12531](#12531))
- Allow unused `# type: ignore` comments in bleeding edge CI jobs. ([\#12576](#12576))
- Remove redundant lines of config from `mypy.ini`. ([\#12608](#12608))
- Update to mypy 0.950. ([\#12650](#12650))
- Use `Concatenate` to better annotate `_do_execute`. ([\#12666](#12666))
- Use `ParamSpec` to refine type hints. ([\#12667](#12667))
- Fix mypy against latest pillow stubs. ([\#12671](#12671))
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Jun 16, 2022
Upstream NEWS, less bugfixes and minor improvements:

Synapse 1.61.0 (2022-06-14)
===========================

This release removes support for the non-standard feature known both as 'groups' and as 'communities', which have been superseded by *Spaces*.

Synapse 1.61.0rc1 (2022-06-07)
==============================

Features
--------

- Add new `media_retention` options to the homeserver config for routinely cleaning up non-recently accessed media. ([\#12732](matrix-org/synapse#12732), [\#12972](matrix-org/synapse#12972), [\#12977](matrix-org/synapse#12977))
- Experimental support for [MSC3772](matrix-org/matrix-spec-proposals#3772): Push rule for mutually related events. ([\#12740](matrix-org/synapse#12740), [\#12859](matrix-org/synapse#12859))
- Update to the `check_event_for_spam` module callback: Deprecate the current callback signature, replace it with a new signature that is both less ambiguous (replacing booleans with explicit allow/block) and more powerful (ability to return explicit error codes). ([\#12808](matrix-org/synapse#12808))
- Add storage and module API methods to get monthly active users (and their corresponding appservices) within an optionally specified time range. ([\#12838](matrix-org/synapse#12838), [\#12917](matrix-org/synapse#12917))
- Support the new error code `ORG.MATRIX.MSC3823.USER_ACCOUNT_SUSPENDED` from [MSC3823](matrix-org/matrix-spec-proposals#3823). ([\#12845](matrix-org/synapse#12845), [\#12923](matrix-org/synapse#12923))
- Add a configurable background job to delete stale devices. ([\#12855](matrix-org/synapse#12855))
- Allow updating a user's password using the admin API without logging out their devices. Contributed by @jcgruenhage. ([\#12952](matrix-org/synapse#12952))


Deprecations and Removals
-------------------------

- Remove support for the non-standard groups/communities feature from Synapse. ([\#12553](matrix-org/synapse#12553), [\#12558](matrix-org/synapse#12558), [\#12563](matrix-org/synapse#12563), [\#12895](matrix-org/synapse#12895), [\#12897](matrix-org/synapse#12897), [\#12899](matrix-org/synapse#12899), [\#12900](matrix-org/synapse#12900), [\#12936](matrix-org/synapse#12936), [\#12966](matrix-org/synapse#12966))
- Remove contributed `kick_users.py` script. This is broken under Python 3, and is not added to the environment when `pip install`ing Synapse. ([\#12908](matrix-org/synapse#12908))



Synapse 1.60.0 (2022-05-31)
===========================

This release of Synapse adds a unique index to the `state_group_edges` table, in
order to prevent accidentally introducing duplicate information (for example,
because a database backup was restored multiple times). If your Synapse database
already has duplicate rows in this table, this could fail with an error and
require manual remediation.

Additionally, the signature of the `check_event_for_spam` module callback has changed.
The previous signature has been deprecated and remains working for now. Module authors
should update their modules to use the new signature where possible.


Synapse 1.60.0rc2 (2022-05-27)
==============================

Features
--------

- Add an option allowing users to use their password to reauthenticate for privileged actions even though password login is disabled. ([\#12883](matrix-org/synapse#12883))


Synapse 1.60.0rc1 (2022-05-24)
==============================

Features
--------

- Measure the time taken in spam-checking callbacks and expose those measurements as metrics. ([\#12513](matrix-org/synapse#12513))
- Add a `default_power_level_content_override` config option to set default room power levels per room preset. ([\#12618](matrix-org/synapse#12618))
- Add support for [MSC3787: Allowing knocks to restricted rooms](matrix-org/matrix-spec-proposals#3787). ([\#12623](matrix-org/synapse#12623))
- Send `USER_IP` commands on a different Redis channel, in order to reduce traffic to workers that do not process these commands. ([\#12672](matrix-org/synapse#12672), [\#12809](matrix-org/synapse#12809))
- Synapse will now reload [cache config](https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html#caching) when it receives a [SIGHUP](https://en.wikipedia.org/wiki/SIGHUP) signal. ([\#12673](matrix-org/synapse#12673))
- Add a config options to allow for auto-tuning of caches. ([\#12701](matrix-org/synapse#12701))
- Update [MSC2716](matrix-org/matrix-spec-proposals#2716) implementation to process marker events from the current state to avoid markers being lost in timeline gaps for federated servers which would cause the imported history to be undiscovered. ([\#12718](matrix-org/synapse#12718))
- Add a `drop_federated_event` callback to `SpamChecker` to disregard inbound federated events before they take up much processing power, in an emergency. ([\#12744](matrix-org/synapse#12744))
- Implement [MSC3818: Copy room type on upgrade](matrix-org/matrix-spec-proposals#3818). ([\#12786](matrix-org/synapse#12786), [\#12792](matrix-org/synapse#12792))
- Update to the `check_event_for_spam` module callback. Deprecate the current callback signature, replace it with a new signature that is both less ambiguous (replacing booleans with explicit allow/block) and more powerful (ability to return explicit error codes). ([\#12808](matrix-org/synapse#12808))


Deprecations and Removals
-------------------------

- Require a body in POST requests to `/rooms/{roomId}/receipt/{receiptType}/{eventId}`, as required by the [Matrix specification](https://spec.matrix.org/v1.2/client-server-api/#post_matrixclientv3roomsroomidreceiptreceipttypeeventid). This breaks compatibility with Element Android 1.2.0 and earlier: users of those clients will be unable to send read receipts. ([\#12709](matrix-org/synapse#12709))



Synapse 1.59.1 (2022-05-18)
===========================

This release fixes a long-standing issue which could prevent Synapse's user directory for updating properly.


Synapse 1.59.0 (2022-05-17)
===========================

Synapse 1.59 makes several changes that server administrators should be aware of:

- Device name lookup over federation is now disabled by default. ([\#12616](matrix-org/synapse#12616))
- The `synapse.app.appservice` and `synapse.app.user_dir` worker application types are now deprecated. ([\#12452](matrix-org/synapse#12452), [\#12654](matrix-org/synapse#12654))

Additionally, this release removes the non-standard `m.login.jwt` login type from Synapse. It can be replaced with `org.matrix.login.jwt` for identical behaviour. This is only used if `jwt_config.enabled` is set to `true` in the configuration. ([\#12597](matrix-org/synapse#12597))


Synapse 1.59.0rc2 (2022-05-16)
==============================


Synapse 1.59.0rc1 (2022-05-10)
==============================

Features
--------

- Support [MSC3266](matrix-org/matrix-spec-proposals#3266) room summaries over federation. ([\#11507](matrix-org/synapse#11507))
- Implement [changes](matrix-org/matrix-spec-proposals@4a77139) to [MSC2285 (hidden read receipts)](matrix-org/matrix-spec-proposals#2285). Contributed by @SimonBrandner. ([\#12168](matrix-org/synapse#12168), [\#12635](matrix-org/synapse#12635), [\#12636](matrix-org/synapse#12636), [\#12670](matrix-org/synapse#12670))
- Extend the [module API](https://github.com/matrix-org/synapse/blob/release-v1.59/synapse/module_api/__init__.py) to allow modules to change actions for existing push rules of local users. ([\#12406](matrix-org/synapse#12406))
- Add the `notify_appservices_from_worker` configuration option (superseding `notify_appservices`) to allow a generic worker to be designated as the worker to send traffic to Application Services. ([\#12452](matrix-org/synapse#12452))
- Add the `update_user_directory_from_worker` configuration option (superseding `update_user_directory`) to allow a generic worker to be designated as the worker to update the user directory. ([\#12654](matrix-org/synapse#12654))
- Add new `enable_registration_token_3pid_bypass` configuration option to allow registrations via token as an alternative to verifying a 3pid. ([\#12526](matrix-org/synapse#12526))
- Implement [MSC3786](matrix-org/matrix-spec-proposals#3786): Add a default push rule to ignore `m.room.server_acl` events. ([\#12601](matrix-org/synapse#12601))
- Add new `mau_appservice_trial_days` configuration option to specify a different trial period for users registered via an appservice. ([\#12619](matrix-org/synapse#12619))


Deprecations and Removals
-------------------------

- Remove unstable identifiers from [MSC3069](matrix-org/matrix-spec-proposals#3069). ([\#12596](matrix-org/synapse#12596))
- Remove the unspecified `m.login.jwt` login type and the unstable `uk.half-shot.msc2778.login.application_service` from
  [MSC2778](matrix-org/matrix-spec-proposals#2778). ([\#12597](matrix-org/synapse#12597))
- Synapse now requires at least Python 3.7.1 (up from 3.7.0), for compatibility with the latest Twisted trunk. ([\#12613](matrix-org/synapse#12613))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

5 participants