Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. #16759

Merged
merged 12 commits into from
Jan 8, 2024
Merged

Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. #16759

merged 12 commits into from
Jan 8, 2024

Conversation

reivilibre
Copy link
Contributor

The idea here being that the directory server shouldn't advertise rooms to a requesting server is the requesting server would not be allowed to join or participate in the room.

Base: develop

Original commit schedule, with full messages:

  1. Pass from_federation_origin down into room list retrieval code

  2. Don't cache /publicRooms response for inbound federated requests

  3. fixup! Don't cache /publicRooms response for inbound federated requests

  4. Cap the number of /publicRooms entries to 100

  5. Simplify code now that you can't request unlimited rooms

  6. Filter out rooms from federated requests that don't have the correct ACL

  7. Request a handful more when filtering ACLs so that we can try to avoid shortchanging the requester

@CLAassistant

This comment was marked as resolved.

Sign in to view
@reivilibre reivilibre marked this pull request as ready for review January 4, 2024 13:47
@reivilibre reivilibre requested a review from a team as a code owner January 4, 2024 13:47
erikjohnston
erikjohnston reviewed Jan 4, 2024
View reviewed changes
synapse/handlers/room_list.py Outdated
# We explicitly don't bother caching searches or requests for
# appservice specific lists.
# We also don't bother caching requests from federated homeservers.
logger.info("Bypassing cache as search request.")
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Let's update this log line too

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Done. Have also demoted to debug since it seems a bit inappropriate for info, but let me know if you disagree

synapse/handlers/room_list.py Outdated
# So instead we re-use the room ID from last time but make the
# bound inclusive, by tacking on a 0x01 byte at the end
# (s+"\x00" is the first string following s, but we can't use "\x00"
# in Postgres, so use "\x01" anyway.)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is this an existing bug or is this somehow caused by the new filtering? I'm somewhat failing to see why we need to add this

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think this is an existing bug. I hit it during testing and thought it was suspicious enough it was worth fixing

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Ah, right. If first_considered_room is None then doesn't that mean that we have reached the end of the list and there are no more rooms? And if so then can we not reuse since_token for the bound?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

If the previous chunk had rooms !A, !B, !C then (from memory) you get a next_chunk token saying !C. The query that follows for the next request has a bound of > !C.

To go back to the prior chunk again, you need the query to have a bound of < !D. But if we have an empty chunk, we don't have any known room ID to stand in for !D, hence the idea of cheekily constructing a fake one "!C\x01"...

If we give the since_token of !C back, then going back to the prior chunk will only return !A, !B as rooms.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh right, yes I see. Has to be said that I'm not a huge fan of just chucking a byte on the end, but I suppose it works.

Can we use the RoomListNextBatch(0, "") since we know we're at the end? Or maybe we should always use the since_token for the prev batch and change the equality bounds on the DB query to avoid all this?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

those are a bit cleverer, yeah. RoomListNextBatch(0, "") sounds like a reasonable idea to me. I think flipping the bounds would also have been fine but I know there is scope to mess that up fairly easily...

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yup fair. I think your solution would have worked fine, but might just cause confusion having invalid room IDs in it, I dunno 🤷 Thanks for changing

erikjohnston
erikjohnston approved these changes Jan 8, 2024
View reviewed changes
Copy link
Member

@erikjohnston erikjohnston left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Woo! Thanks

@reivilibre reivilibre merged commit a83a337 into develop Jan 8, 2024
39 checks passed
@reivilibre reivilibre deleted the rei/acl_publicrooms branch January 8, 2024 17:24
yingziwu added a commit to yingziwu/synapse that referenced this pull request Jan 16, 2024
@yingziwu
Merge tag 'v1.99.0' into develop
d2a3f9b 
Synapse 1.99.0 is the first Synapse release under an AGPLv3.0 licence (with CLA to enable Element to sell AGPL
exceptions). You can read more about this here:

 - https://matrix.org/blog/2023/11/06/future-of-synapse-dendrite/
 - https://element.io/blog/element-to-adopt-agplv3/
 - https://element.io/blog/synapse-now-lives-at-github-com-element-hq-synapse/

No significant changes since 1.99.0rc1.

- Add [config options](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#server_notices) to set the avatar and the topic of the server notices room, as well as the avatar of the server notices user. ([\matrix-org#16679](matrix-org#16679))
- Add config option [`email.notif_delay_before_mail`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#email) to tweak the delay before an email is sent following a notification. ([\matrix-org#16696](matrix-org#16696))
- Add new configuration option [`sentry.environment`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#sentry) for improved system monitoring. Contributed by @zeeshanrafiqrana. ([\matrix-org#16738](matrix-org#16738))
- Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. ([\matrix-org#16759](element-hq/synapse#16759))

- Fix a long-standing bug where the signing keys generated by Synapse were world-readable. Contributed by Fabian Klemp. ([\matrix-org#16740](matrix-org#16740))
- Fix email verification redirection. Contributed by Fadhlan Ridhwanallah. ([\matrix-org#16761](element-hq/synapse#16761))
- Fixed a bug that prevented users from being queried by display name if it contains non-ASCII characters. ([\matrix-org#16767](element-hq/synapse#16767))
- Allow reactivate user without password with Admin API in some edge cases. ([\matrix-org#16770](element-hq/synapse#16770))
- Adds the `recursion_depth` parameter to the response of the /relations endpoint if MSC3981 recursion is being performed. ([\#16775](element-hq/synapse#16775))

- Added version picker for Synapse documentation. Contributed by @Dmytro27Ind. ([\matrix-org#16533](matrix-org#16533))
- Clarify that `password_config.enabled: "only_for_reauth"` does not allow new logins to be created using password auth. ([\matrix-org#16737](matrix-org#16737))
- Remove value from header in configuration documentation for `refresh_token_lifetime`. ([\matrix-org#16763](element-hq/synapse#16763))
- Add another custom statistics collection server to the documentation. Contributed by @loelkes. ([\matrix-org#16769](element-hq/synapse#16769))

- Remove run-once workflow after adding the version picker to the documentation. ([\matrix-org#9453](element-hq/synapse#9453))
- Update the implementation of [MSC2965](matrix-org/matrix-spec-proposals#2965) (OIDC Provider discovery). ([\matrix-org#16726](matrix-org#16726))
- Move the rust stubs inline for better IDE integration. ([\matrix-org#16757](element-hq/synapse#16757))
- Fix sample config doc CI. ([\matrix-org#16758](element-hq/synapse#16758))
- Simplify event internal metadata class. ([\matrix-org#16762](element-hq/synapse#16762), [\#16780](element-hq/synapse#16780))
- Sign the published docker image using [cosign](https://docs.sigstore.dev/). ([\#16774](element-hq/synapse#16774))
- Port `EventInternalMetadata` class to Rust. ([\#16782](element-hq/synapse#16782))

* Bump actions/setup-go from 4 to 5. ([\matrix-org#16749](matrix-org#16749))
* Bump actions/setup-python from 4 to 5. ([\matrix-org#16748](matrix-org#16748))
* Bump immutabledict from 3.0.0 to 4.0.0. ([\matrix-org#16743](matrix-org#16743))
* Bump isort from 5.12.0 to 5.13.0. ([\matrix-org#16745](matrix-org#16745))
* Bump isort from 5.13.0 to 5.13.1. ([\matrix-org#16752](matrix-org#16752))
* Bump pydantic from 2.5.1 to 2.5.2. ([\matrix-org#16747](matrix-org#16747))
* Bump ruff from 0.1.6 to 0.1.7. ([\matrix-org#16746](matrix-org#16746))
* Bump types-setuptools from 68.2.0.2 to 69.0.0.0. ([\matrix-org#16744](matrix-org#16744))
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Mar 3, 2024
@Midar
Update chat/matrix-synapse to 1.101.0
0b2ea45 
!!! THIS CHANGES THE LICENSE TO AGPLv3 !!!


# Synapse 1.101.0 (2024-02-13)

### Bugfixes

- Fix performance regression when fetching auth chains from the DB. Introduced in v1.100.0. ([\#16893](element-hq/synapse#16893))




# Synapse 1.101.0rc1 (2024-02-06)

### Improved Documentation

- Fix broken links in the documentation. ([\#16853](element-hq/synapse#16853))
- Update MacOS installation instructions to mention that libicu is optional. ([\#16854](element-hq/synapse#16854))
- The version picker now correctly lists versions after `v1.98.0`. ([\#16880](element-hq/synapse#16880))

### Internal Changes

- Add support for stabilised [MSC3981](matrix-org/matrix-spec-proposals#3981) that adds a `recurse` parameter on the `/relations` API. ([\#16842](element-hq/synapse#16842))



### Updates to locked dependencies

* Bump dorny/paths-filter from 2 to 3. ([\#16869](element-hq/synapse#16869))
* Bump gitpython from 3.1.40 to 3.1.41. ([\#16850](element-hq/synapse#16850))
* Bump hiredis from 2.2.3 to 2.3.2. ([\#16862](element-hq/synapse#16862))
* Bump jsonschema from 4.20.0 to 4.21.1. ([\#16887](element-hq/synapse#16887))
* Bump lxml-stubs from 0.4.0 to 0.5.1. ([\#16885](element-hq/synapse#16885))
* Bump mypy-zope from 1.0.1 to 1.0.3. ([\#16865](element-hq/synapse#16865))
* Bump phonenumbers from 8.13.26 to 8.13.29. ([\#16868](element-hq/synapse#16868))
* Bump pydantic from 2.5.3 to 2.6.0. ([\#16888](element-hq/synapse#16888))
* Bump sentry-sdk from 1.39.1 to 1.40.0. ([\#16889](element-hq/synapse#16889))
* Bump serde from 1.0.195 to 1.0.196. ([\#16867](element-hq/synapse#16867))
* Bump serde_json from 1.0.111 to 1.0.113. ([\#16866](element-hq/synapse#16866))
* Bump sigstore/cosign-installer from 3.3.0 to 3.4.0. ([\#16890](element-hq/synapse#16890))
* Bump types-pillow from 10.1.0.2 to 10.2.0.20240125. ([\#16864](element-hq/synapse#16864))
* Bump types-requests from 2.31.0.10 to 2.31.0.20240125. ([\#16886](element-hq/synapse#16886))
* Bump types-setuptools from 69.0.0.0 to 69.0.0.20240125. ([\#16863](element-hq/synapse#16863))

# Synapse 1.100.0 (2024-01-30)

No significant changes since 1.100.0rc3.




# Synapse 1.100.0rc3 (2024-01-24)

### Bugfixes

- Fix database performance regression due to changing Postgres table statistics. Introduced in v1.100.0rc1. ([\#16849](element-hq/synapse#16849))




# Synapse 1.100.0rc2 (2024-01-24)

This version is the same as 1.100.0rc1 but with fixes to the release process.

### Internal Changes

- Downgrade the `download-artifact` and `upload-artifact` actions to v3 due to breaking changes. ([\#16847](element-hq/synapse#16847))


# Synapse 1.100.0rc1 (2024-01-23)

*This version was never released to PyPI or the Debian repository due to failures in the automatic part of the release process.*

### Features

- Advertise experimental support for [MSC4028](matrix-org/matrix-spec-proposals#4028) through `/_matrix/clients/versions` if enabled. Contributed by @hanadi92. ([\#16787](element-hq/synapse#16787))

### Bugfixes

- Handle wildcard type filters properly for room messages endpoint. Contributed by Mo Balaa. ([\#14984](element-hq/synapse#14984))

### Improved Documentation

- Add a link to the "Request log format" explainer on the "Logging sample config" documentation page. ([\#16778](element-hq/synapse#16778))
- Fix broken links in issue templates and documentation. ([\#16810](element-hq/synapse#16810))
- NGINX listen http2 deprecation in documentation template for reverse proxy. ([\#16831](element-hq/synapse#16831))

### Internal Changes

- Faster partial join to room with complex auth graph. ([\#7](element-hq/synapse#7))
- Improve DB performance of calculating badge counts for push. ([\#16756](element-hq/synapse#16756))
- Split up deleting devices into batches. ([\#16766](element-hq/synapse#16766))
- Remove CI check for sign-off as we require a CLA signature instead. ([\#16776](element-hq/synapse#16776))
- Ensure CI fails when linting fails to make sure auto-merge does the correct thing. ([\#16781](element-hq/synapse#16781))
- Faster load recents for sync by reducing amount of state pulled out. ([\#16783](element-hq/synapse#16783))
- Reduce amount of state pulled out when querying federation hierachy. ([\#16785](element-hq/synapse#16785))
- Pull less state out of the DB when we retry fetching old events during backfill. ([\#16788](element-hq/synapse#16788))
- Optimize query for fetching to-device messages in `/sync`. ([\#16805](element-hq/synapse#16805))
- Reject OIDC config when `client_secret` isn't specified, but the auth method requires one. ([\#16806](element-hq/synapse#16806))
- Allow room creation but not publishing to continue if room publication rules are violated when creating
  a new room. ([\#16811](element-hq/synapse#16811))
- Bump minimum supported Rust version to 1.65.0. ([\#16818](element-hq/synapse#16818))
- Fixup copyright lines in file headers after the licensing change. ([\#16820](element-hq/synapse#16820))
- Add a `--generate-only` option to the internal configuration/launch script for Complement. ([\#16828](element-hq/synapse#16828))
- Preparatory work for tweaking performance of auth chain lookups. ([\#16833](element-hq/synapse#16833))
- Speed up e2e device keys queries for bot accounts. ([\#16841](element-hq/synapse#16841))

### Updates to locked dependencies

* Bump actions/cache from 3 to 4. ([\#16832](element-hq/synapse#16832))
* Bump actions/download-artifact from 3 to 4. ([\#16795](element-hq/synapse#16795))
* Bump actions/upload-artifact from 3 to 4. ([\#16796](element-hq/synapse#16796))
* Bump anyhow from 1.0.75 to 1.0.79. ([\#16789](element-hq/synapse#16789))
* Bump authlib from 1.2.1 to 1.3.0. ([\#16801](element-hq/synapse#16801))
* Bump dawidd6/action-download-artifact from 2.28.0 to 3.0.0. ([\#16794](element-hq/synapse#16794))
* Bump immutabledict from 4.0.0 to 4.1.0. ([\#16812](element-hq/synapse#16812))
* Bump isort from 5.13.1 to 5.13.2. ([\#16835](element-hq/synapse#16835))
* Bump lxml from 4.9.3 to 5.1.0. ([\#16813](element-hq/synapse#16813))
* Bump pillow from 10.1.0 to 10.2.0. ([\#16802](element-hq/synapse#16802))
* Bump pydantic from 2.5.2 to 2.5.3. ([\#16836](element-hq/synapse#16836))
* Bump pyo3 from 0.20.0 to 0.20.2. ([\#16791](element-hq/synapse#16791))
* Bump regex from 1.9.6 to 1.10.3. ([\#16837](element-hq/synapse#16837))
* Bump ruff from 0.1.13 to 0.1.14. ([\#16838](element-hq/synapse#16838))
* Bump ruff from 0.1.7 to 0.1.13. ([\#16814](element-hq/synapse#16814))
* Bump sentry-sdk from 1.35.0 to 1.39.1. ([\#16799](element-hq/synapse#16799))
* Bump serde_json from 1.0.108 to 1.0.111. ([\#16792](element-hq/synapse#16792))
* Bump service-identity from 23.1.0 to 24.1.0. ([\#16816](element-hq/synapse#16816))
* Bump types-commonmark from 0.9.2.4 to 0.9.2.20240106. ([\#16797](element-hq/synapse#16797))
* Bump types-jsonschema from 4.20.0.0 to 4.20.0.20240105. ([\#16800](element-hq/synapse#16800))
* Bump types-jsonschema from 4.20.0.20240105 to 4.21.0.20240118. ([\#16834](element-hq/synapse#16834))
* Bump types-netaddr from 0.9.0.1 to 0.10.0.20240106. ([\#16839](element-hq/synapse#16839))
* Bump typing-extensions from 4.8.0 to 4.9.0. ([\#16815](element-hq/synapse#16815))


# Synapse 1.99.0 (2024-01-16)

Synapse 1.99.0 is the first Synapse release under an AGPLv3.0 licence (with CLA to enable Element to sell AGPL
exceptions). You can read more about this here:

 - https://matrix.org/blog/2023/11/06/future-of-synapse-dendrite/
 - https://element.io/blog/element-to-adopt-agplv3/
 - https://element.io/blog/synapse-now-lives-at-github-com-element-hq-synapse/

No significant changes since 1.99.0rc1.


# Synapse 1.99.0rc1 (2024-01-09)

### Features

- Add [config options](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#server_notices) to set the avatar and the topic of the server notices room, as well as the avatar of the server notices user. ([\#16679](matrix-org/synapse#16679))
- Add config option [`email.notif_delay_before_mail`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#email) to tweak the delay before an email is sent following a notification. ([\#16696](matrix-org/synapse#16696))
- Add new configuration option [`sentry.environment`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#sentry) for improved system monitoring. Contributed by @zeeshanrafiqrana. ([\#16738](matrix-org/synapse#16738))
- Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. ([\#16759](element-hq/synapse#16759))

### Bugfixes

- Fix a long-standing bug where the signing keys generated by Synapse were world-readable. Contributed by Fabian Klemp. ([\#16740](matrix-org/synapse#16740))
- Fix email verification redirection. Contributed by Fadhlan Ridhwanallah. ([\#16761](element-hq/synapse#16761))
- Fixed a bug that prevented users from being queried by display name if it contains non-ASCII characters. ([\#16767](element-hq/synapse#16767))
- Allow reactivate user without password with Admin API in some edge cases. ([\#16770](element-hq/synapse#16770))
- Adds the `recursion_depth` parameter to the response of the /relations endpoint if MSC3981 recursion is being performed. ([\#16775](element-hq/synapse#16775))

### Improved Documentation

- Added version picker for Synapse documentation. Contributed by @Dmytro27Ind. ([\#16533](matrix-org/synapse#16533))
- Clarify that `password_config.enabled: "only_for_reauth"` does not allow new logins to be created using password auth. ([\#16737](matrix-org/synapse#16737))
- Remove value from header in configuration documentation for `refresh_token_lifetime`. ([\#16763](element-hq/synapse#16763))
- Add another custom statistics collection server to the documentation. Contributed by @loelkes. ([\#16769](element-hq/synapse#16769))

### Internal Changes

- Remove run-once workflow after adding the version picker to the documentation. ([\#9453](element-hq/synapse#9453))
- Update the implementation of [MSC2965](matrix-org/matrix-spec-proposals#2965) (OIDC Provider discovery). ([\#16726](matrix-org/synapse#16726))
- Move the rust stubs inline for better IDE integration. ([\#16757](element-hq/synapse#16757))
- Fix sample config doc CI. ([\#16758](element-hq/synapse#16758))
- Simplify event internal metadata class. ([\#16762](element-hq/synapse#16762), [\#16780](element-hq/synapse#16780))
- Sign the published docker image using [cosign](https://docs.sigstore.dev/). ([\#16774](element-hq/synapse#16774))
- Port `EventInternalMetadata` class to Rust. ([\#16782](element-hq/synapse#16782))



### Updates to locked dependencies

* Bump actions/setup-go from 4 to 5. ([\#16749](matrix-org/synapse#16749))
* Bump actions/setup-python from 4 to 5. ([\#16748](matrix-org/synapse#16748))
* Bump immutabledict from 3.0.0 to 4.0.0. ([\#16743](matrix-org/synapse#16743))
* Bump isort from 5.12.0 to 5.13.0. ([\#16745](matrix-org/synapse#16745))
* Bump isort from 5.13.0 to 5.13.1. ([\#16752](matrix-org/synapse#16752))
* Bump pydantic from 2.5.1 to 2.5.2. ([\#16747](matrix-org/synapse#16747))
* Bump ruff from 0.1.6 to 0.1.7. ([\#16746](matrix-org/synapse#16746))
* Bump types-setuptools from 68.2.0.2 to 69.0.0.0. ([\#16744](matrix-org/synapse#16744))
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@erikjohnston erikjohnston erikjohnston approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@reivilibre @CLAassistant @erikjohnston