Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Reduce amount of state pulled out when querying federation hierachy #16785

Merged
merged 6 commits into from
Jan 10, 2024
Merged

Reduce amount of state pulled out when querying federation hierachy #16785

merged 6 commits into from
Jan 10, 2024

Conversation

erikjohnston
Copy link
Member

There are two changes here:

  1. Only pull out the required state when handling the request.
  2. Change the get filtered state return type to check that we're only querying state that was requested

@erikjohnston erikjohnston force-pushed the erikj/less_state_restricted_check branch from d1adcb6 to 146ffee Compare January 8, 2024 11:07
@erikjohnston erikjohnston marked this pull request as ready for review January 8, 2024 15:29
@erikjohnston erikjohnston requested a review from a team as a code owner January 8, 2024 15:29
reivilibre
reivilibre approved these changes Jan 10, 2024
View reviewed changes
Copy link
Contributor

@reivilibre reivilibre left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

SGTM overall, but think we should be a bit stricter about runtime type errors rather than shrugging them off as return False unless I'm missing something.

synapse/storage/databases/main/state.py
Comment on lines +738 to +744
class StateMapWrapper(Dict[StateKey, str]):
"""A wrapper around a StateMap[str] to ensure that we only query for items
that were not filtered out.

This is to help prevent bugs where we filter out state but other bits of the
code expect the state to be there.
"""
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I like it.

synapse/types/state.py Outdated Show resolved Hide resolved
synapse/types/state.py Outdated Show resolved Hide resolved
erikjohnston and others added 2 commits January 10, 2024 14:05
@erikjohnston @reivilibre
Apply suggestions from code review
24cb886 
Co-authored-by: reivilibre <oliverw@matrix.org>
@erikjohnston
liniting
183202b
@erikjohnston erikjohnston enabled auto-merge (squash) January 10, 2024 14:14
@erikjohnston erikjohnston merged commit 578c5c7 into develop Jan 10, 2024
38 checks passed
@erikjohnston erikjohnston deleted the erikj/less_state_restricted_check branch January 10, 2024 14:31
yingziwu added a commit to yingziwu/synapse that referenced this pull request Feb 1, 2024
@yingziwu
Merge tag 'v1.100.0' into develop
1eca755 
No significant changes since 1.100.0rc3.

- Fix database performance regression due to changing Postgres table statistics. Introduced in v1.100.0rc1. ([\#16849](element-hq/synapse#16849))

This version is the same as 1.100.0rc1 but with fixes to the release process.

- Downgrade the `download-artifact` and `upload-artifact` actions to v3 due to breaking changes. ([\#16847](element-hq/synapse#16847))

*This version was never released to PyPI or the Debian repository due to failures in the automatic part of the release process.*

- Advertise experimental support for [MSC4028](matrix-org/matrix-spec-proposals#4028) through `/_matrix/clients/versions` if enabled. Contributed by @hanadi92. ([\#16787](element-hq/synapse#16787))

- Handle wildcard type filters properly for room messages endpoint. Contributed by Mo Balaa. ([\matrix-org#14984](element-hq/synapse#14984))

- Add a link to the "Request log format" explainer on the "Logging sample config" documentation page. ([\#16778](element-hq/synapse#16778))
- Fix broken links in issue templates and documentation. ([\#16810](element-hq/synapse#16810))
- NGINX listen http2 deprecation in documentation template for reverse proxy. ([\#16831](element-hq/synapse#16831))

- Faster partial join to room with complex auth graph. ([\matrix-org#7](element-hq/synapse#7))
- Improve DB performance of calculating badge counts for push. ([\matrix-org#16756](element-hq/synapse#16756))
- Split up deleting devices into batches. ([\matrix-org#16766](element-hq/synapse#16766))
- Remove CI check for sign-off as we require a CLA signature instead. ([\#16776](element-hq/synapse#16776))
- Ensure CI fails when linting fails to make sure auto-merge does the correct thing. ([\#16781](element-hq/synapse#16781))
- Faster load recents for sync by reducing amount of state pulled out. ([\#16783](element-hq/synapse#16783))
- Reduce amount of state pulled out when querying federation hierachy. ([\#16785](element-hq/synapse#16785))
- Pull less state out of the DB when we retry fetching old events during backfill. ([\#16788](element-hq/synapse#16788))
- Optimize query for fetching to-device messages in `/sync`. ([\#16805](element-hq/synapse#16805))
- Reject OIDC config when `client_secret` isn't specified, but the auth method requires one. ([\#16806](element-hq/synapse#16806))
- Allow room creation but not publishing to continue if room publication rules are violated when creating
  a new room. ([\#16811](element-hq/synapse#16811))
- Bump minimum supported Rust version to 1.65.0. ([\#16818](element-hq/synapse#16818))
- Fixup copyright lines in file headers after the licensing change. ([\#16820](element-hq/synapse#16820))
- Add a `--generate-only` option to the internal configuration/launch script for Complement. ([\#16828](element-hq/synapse#16828))
- Preparatory work for tweaking performance of auth chain lookups. ([\#16833](element-hq/synapse#16833))
- Speed up e2e device keys queries for bot accounts. ([\#16841](element-hq/synapse#16841))

* Bump actions/cache from 3 to 4. ([\#16832](element-hq/synapse#16832))
* Bump actions/download-artifact from 3 to 4. ([\#16795](element-hq/synapse#16795))
* Bump actions/upload-artifact from 3 to 4. ([\#16796](element-hq/synapse#16796))
* Bump anyhow from 1.0.75 to 1.0.79. ([\#16789](element-hq/synapse#16789))
* Bump authlib from 1.2.1 to 1.3.0. ([\#16801](element-hq/synapse#16801))
* Bump dawidd6/action-download-artifact from 2.28.0 to 3.0.0. ([\#16794](element-hq/synapse#16794))
* Bump immutabledict from 4.0.0 to 4.1.0. ([\#16812](element-hq/synapse#16812))
* Bump isort from 5.13.1 to 5.13.2. ([\#16835](element-hq/synapse#16835))
* Bump lxml from 4.9.3 to 5.1.0. ([\#16813](element-hq/synapse#16813))
* Bump pillow from 10.1.0 to 10.2.0. ([\#16802](element-hq/synapse#16802))
* Bump pydantic from 2.5.2 to 2.5.3. ([\#16836](element-hq/synapse#16836))
* Bump pyo3 from 0.20.0 to 0.20.2. ([\#16791](element-hq/synapse#16791))
* Bump regex from 1.9.6 to 1.10.3. ([\#16837](element-hq/synapse#16837))
* Bump ruff from 0.1.13 to 0.1.14. ([\#16838](element-hq/synapse#16838))
* Bump ruff from 0.1.7 to 0.1.13. ([\#16814](element-hq/synapse#16814))
* Bump sentry-sdk from 1.35.0 to 1.39.1. ([\#16799](element-hq/synapse#16799))
* Bump serde_json from 1.0.108 to 1.0.111. ([\#16792](element-hq/synapse#16792))
* Bump service-identity from 23.1.0 to 24.1.0. ([\#16816](element-hq/synapse#16816))
* Bump types-commonmark from 0.9.2.4 to 0.9.2.20240106. ([\#16797](element-hq/synapse#16797))
* Bump types-jsonschema from 4.20.0.0 to 4.20.0.20240105. ([\#16800](element-hq/synapse#16800))
* Bump types-jsonschema from 4.20.0.20240105 to 4.21.0.20240118. ([\#16834](element-hq/synapse#16834))
* Bump types-netaddr from 0.9.0.1 to 0.10.0.20240106. ([\#16839](element-hq/synapse#16839))
* Bump typing-extensions from 4.8.0 to 4.9.0. ([\#16815](element-hq/synapse#16815))
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Mar 3, 2024
@Midar
Update chat/matrix-synapse to 1.101.0
0b2ea45 
!!! THIS CHANGES THE LICENSE TO AGPLv3 !!!


# Synapse 1.101.0 (2024-02-13)

### Bugfixes

- Fix performance regression when fetching auth chains from the DB. Introduced in v1.100.0. ([\#16893](element-hq/synapse#16893))




# Synapse 1.101.0rc1 (2024-02-06)

### Improved Documentation

- Fix broken links in the documentation. ([\#16853](element-hq/synapse#16853))
- Update MacOS installation instructions to mention that libicu is optional. ([\#16854](element-hq/synapse#16854))
- The version picker now correctly lists versions after `v1.98.0`. ([\#16880](element-hq/synapse#16880))

### Internal Changes

- Add support for stabilised [MSC3981](matrix-org/matrix-spec-proposals#3981) that adds a `recurse` parameter on the `/relations` API. ([\#16842](element-hq/synapse#16842))



### Updates to locked dependencies

* Bump dorny/paths-filter from 2 to 3. ([\#16869](element-hq/synapse#16869))
* Bump gitpython from 3.1.40 to 3.1.41. ([\#16850](element-hq/synapse#16850))
* Bump hiredis from 2.2.3 to 2.3.2. ([\#16862](element-hq/synapse#16862))
* Bump jsonschema from 4.20.0 to 4.21.1. ([\#16887](element-hq/synapse#16887))
* Bump lxml-stubs from 0.4.0 to 0.5.1. ([\#16885](element-hq/synapse#16885))
* Bump mypy-zope from 1.0.1 to 1.0.3. ([\#16865](element-hq/synapse#16865))
* Bump phonenumbers from 8.13.26 to 8.13.29. ([\#16868](element-hq/synapse#16868))
* Bump pydantic from 2.5.3 to 2.6.0. ([\#16888](element-hq/synapse#16888))
* Bump sentry-sdk from 1.39.1 to 1.40.0. ([\#16889](element-hq/synapse#16889))
* Bump serde from 1.0.195 to 1.0.196. ([\#16867](element-hq/synapse#16867))
* Bump serde_json from 1.0.111 to 1.0.113. ([\#16866](element-hq/synapse#16866))
* Bump sigstore/cosign-installer from 3.3.0 to 3.4.0. ([\#16890](element-hq/synapse#16890))
* Bump types-pillow from 10.1.0.2 to 10.2.0.20240125. ([\#16864](element-hq/synapse#16864))
* Bump types-requests from 2.31.0.10 to 2.31.0.20240125. ([\#16886](element-hq/synapse#16886))
* Bump types-setuptools from 69.0.0.0 to 69.0.0.20240125. ([\#16863](element-hq/synapse#16863))

# Synapse 1.100.0 (2024-01-30)

No significant changes since 1.100.0rc3.




# Synapse 1.100.0rc3 (2024-01-24)

### Bugfixes

- Fix database performance regression due to changing Postgres table statistics. Introduced in v1.100.0rc1. ([\#16849](element-hq/synapse#16849))




# Synapse 1.100.0rc2 (2024-01-24)

This version is the same as 1.100.0rc1 but with fixes to the release process.

### Internal Changes

- Downgrade the `download-artifact` and `upload-artifact` actions to v3 due to breaking changes. ([\#16847](element-hq/synapse#16847))


# Synapse 1.100.0rc1 (2024-01-23)

*This version was never released to PyPI or the Debian repository due to failures in the automatic part of the release process.*

### Features

- Advertise experimental support for [MSC4028](matrix-org/matrix-spec-proposals#4028) through `/_matrix/clients/versions` if enabled. Contributed by @hanadi92. ([\#16787](element-hq/synapse#16787))

### Bugfixes

- Handle wildcard type filters properly for room messages endpoint. Contributed by Mo Balaa. ([\#14984](element-hq/synapse#14984))

### Improved Documentation

- Add a link to the "Request log format" explainer on the "Logging sample config" documentation page. ([\#16778](element-hq/synapse#16778))
- Fix broken links in issue templates and documentation. ([\#16810](element-hq/synapse#16810))
- NGINX listen http2 deprecation in documentation template for reverse proxy. ([\#16831](element-hq/synapse#16831))

### Internal Changes

- Faster partial join to room with complex auth graph. ([\#7](element-hq/synapse#7))
- Improve DB performance of calculating badge counts for push. ([\#16756](element-hq/synapse#16756))
- Split up deleting devices into batches. ([\#16766](element-hq/synapse#16766))
- Remove CI check for sign-off as we require a CLA signature instead. ([\#16776](element-hq/synapse#16776))
- Ensure CI fails when linting fails to make sure auto-merge does the correct thing. ([\#16781](element-hq/synapse#16781))
- Faster load recents for sync by reducing amount of state pulled out. ([\#16783](element-hq/synapse#16783))
- Reduce amount of state pulled out when querying federation hierachy. ([\#16785](element-hq/synapse#16785))
- Pull less state out of the DB when we retry fetching old events during backfill. ([\#16788](element-hq/synapse#16788))
- Optimize query for fetching to-device messages in `/sync`. ([\#16805](element-hq/synapse#16805))
- Reject OIDC config when `client_secret` isn't specified, but the auth method requires one. ([\#16806](element-hq/synapse#16806))
- Allow room creation but not publishing to continue if room publication rules are violated when creating
  a new room. ([\#16811](element-hq/synapse#16811))
- Bump minimum supported Rust version to 1.65.0. ([\#16818](element-hq/synapse#16818))
- Fixup copyright lines in file headers after the licensing change. ([\#16820](element-hq/synapse#16820))
- Add a `--generate-only` option to the internal configuration/launch script for Complement. ([\#16828](element-hq/synapse#16828))
- Preparatory work for tweaking performance of auth chain lookups. ([\#16833](element-hq/synapse#16833))
- Speed up e2e device keys queries for bot accounts. ([\#16841](element-hq/synapse#16841))

### Updates to locked dependencies

* Bump actions/cache from 3 to 4. ([\#16832](element-hq/synapse#16832))
* Bump actions/download-artifact from 3 to 4. ([\#16795](element-hq/synapse#16795))
* Bump actions/upload-artifact from 3 to 4. ([\#16796](element-hq/synapse#16796))
* Bump anyhow from 1.0.75 to 1.0.79. ([\#16789](element-hq/synapse#16789))
* Bump authlib from 1.2.1 to 1.3.0. ([\#16801](element-hq/synapse#16801))
* Bump dawidd6/action-download-artifact from 2.28.0 to 3.0.0. ([\#16794](element-hq/synapse#16794))
* Bump immutabledict from 4.0.0 to 4.1.0. ([\#16812](element-hq/synapse#16812))
* Bump isort from 5.13.1 to 5.13.2. ([\#16835](element-hq/synapse#16835))
* Bump lxml from 4.9.3 to 5.1.0. ([\#16813](element-hq/synapse#16813))
* Bump pillow from 10.1.0 to 10.2.0. ([\#16802](element-hq/synapse#16802))
* Bump pydantic from 2.5.2 to 2.5.3. ([\#16836](element-hq/synapse#16836))
* Bump pyo3 from 0.20.0 to 0.20.2. ([\#16791](element-hq/synapse#16791))
* Bump regex from 1.9.6 to 1.10.3. ([\#16837](element-hq/synapse#16837))
* Bump ruff from 0.1.13 to 0.1.14. ([\#16838](element-hq/synapse#16838))
* Bump ruff from 0.1.7 to 0.1.13. ([\#16814](element-hq/synapse#16814))
* Bump sentry-sdk from 1.35.0 to 1.39.1. ([\#16799](element-hq/synapse#16799))
* Bump serde_json from 1.0.108 to 1.0.111. ([\#16792](element-hq/synapse#16792))
* Bump service-identity from 23.1.0 to 24.1.0. ([\#16816](element-hq/synapse#16816))
* Bump types-commonmark from 0.9.2.4 to 0.9.2.20240106. ([\#16797](element-hq/synapse#16797))
* Bump types-jsonschema from 4.20.0.0 to 4.20.0.20240105. ([\#16800](element-hq/synapse#16800))
* Bump types-jsonschema from 4.20.0.20240105 to 4.21.0.20240118. ([\#16834](element-hq/synapse#16834))
* Bump types-netaddr from 0.9.0.1 to 0.10.0.20240106. ([\#16839](element-hq/synapse#16839))
* Bump typing-extensions from 4.8.0 to 4.9.0. ([\#16815](element-hq/synapse#16815))


# Synapse 1.99.0 (2024-01-16)

Synapse 1.99.0 is the first Synapse release under an AGPLv3.0 licence (with CLA to enable Element to sell AGPL
exceptions). You can read more about this here:

 - https://matrix.org/blog/2023/11/06/future-of-synapse-dendrite/
 - https://element.io/blog/element-to-adopt-agplv3/
 - https://element.io/blog/synapse-now-lives-at-github-com-element-hq-synapse/

No significant changes since 1.99.0rc1.


# Synapse 1.99.0rc1 (2024-01-09)

### Features

- Add [config options](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#server_notices) to set the avatar and the topic of the server notices room, as well as the avatar of the server notices user. ([\#16679](matrix-org/synapse#16679))
- Add config option [`email.notif_delay_before_mail`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#email) to tweak the delay before an email is sent following a notification. ([\#16696](matrix-org/synapse#16696))
- Add new configuration option [`sentry.environment`](https://element-hq.github.io/synapse/v1.99/usage/configuration/config_documentation.html#sentry) for improved system monitoring. Contributed by @zeeshanrafiqrana. ([\#16738](matrix-org/synapse#16738))
- Filter out rooms from the room directory being served to other homeservers when those rooms block that homeserver by their Access Control Lists. ([\#16759](element-hq/synapse#16759))

### Bugfixes

- Fix a long-standing bug where the signing keys generated by Synapse were world-readable. Contributed by Fabian Klemp. ([\#16740](matrix-org/synapse#16740))
- Fix email verification redirection. Contributed by Fadhlan Ridhwanallah. ([\#16761](element-hq/synapse#16761))
- Fixed a bug that prevented users from being queried by display name if it contains non-ASCII characters. ([\#16767](element-hq/synapse#16767))
- Allow reactivate user without password with Admin API in some edge cases. ([\#16770](element-hq/synapse#16770))
- Adds the `recursion_depth` parameter to the response of the /relations endpoint if MSC3981 recursion is being performed. ([\#16775](element-hq/synapse#16775))

### Improved Documentation

- Added version picker for Synapse documentation. Contributed by @Dmytro27Ind. ([\#16533](matrix-org/synapse#16533))
- Clarify that `password_config.enabled: "only_for_reauth"` does not allow new logins to be created using password auth. ([\#16737](matrix-org/synapse#16737))
- Remove value from header in configuration documentation for `refresh_token_lifetime`. ([\#16763](element-hq/synapse#16763))
- Add another custom statistics collection server to the documentation. Contributed by @loelkes. ([\#16769](element-hq/synapse#16769))

### Internal Changes

- Remove run-once workflow after adding the version picker to the documentation. ([\#9453](element-hq/synapse#9453))
- Update the implementation of [MSC2965](matrix-org/matrix-spec-proposals#2965) (OIDC Provider discovery). ([\#16726](matrix-org/synapse#16726))
- Move the rust stubs inline for better IDE integration. ([\#16757](element-hq/synapse#16757))
- Fix sample config doc CI. ([\#16758](element-hq/synapse#16758))
- Simplify event internal metadata class. ([\#16762](element-hq/synapse#16762), [\#16780](element-hq/synapse#16780))
- Sign the published docker image using [cosign](https://docs.sigstore.dev/). ([\#16774](element-hq/synapse#16774))
- Port `EventInternalMetadata` class to Rust. ([\#16782](element-hq/synapse#16782))



### Updates to locked dependencies

* Bump actions/setup-go from 4 to 5. ([\#16749](matrix-org/synapse#16749))
* Bump actions/setup-python from 4 to 5. ([\#16748](matrix-org/synapse#16748))
* Bump immutabledict from 3.0.0 to 4.0.0. ([\#16743](matrix-org/synapse#16743))
* Bump isort from 5.12.0 to 5.13.0. ([\#16745](matrix-org/synapse#16745))
* Bump isort from 5.13.0 to 5.13.1. ([\#16752](matrix-org/synapse#16752))
* Bump pydantic from 2.5.1 to 2.5.2. ([\#16747](matrix-org/synapse#16747))
* Bump ruff from 0.1.6 to 0.1.7. ([\#16746](matrix-org/synapse#16746))
* Bump types-setuptools from 68.2.0.2 to 69.0.0.0. ([\#16744](matrix-org/synapse#16744))
@hakkelt hakkelt mentioned this pull request Jun 6, 2024
Open
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@reivilibre reivilibre reivilibre approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
2 participants
@erikjohnston @reivilibre