Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Allow enabling the asyncio reactor in complement #14858

Merged
merged 14 commits into from
Feb 1, 2023
Merged

Allow enabling the asyncio reactor in complement #14858

merged 14 commits into from
Feb 1, 2023

Conversation

realtyem
Copy link
Contributor

@realtyem realtyem commented Jan 16, 2023 •
edited
Loading

From #14099

This allows enabling the asyncio reactor by setting the ASYNCIO_REACTOR=1 environment variable.
It also adds jobs to the CI to run those in CI.

I understand this is an experiment. I was curious to see where this goes. While I'm sure there's a better way to do the imports, just using a different environment variable circumvents the issue easily for now. If this sticks around, and gets used for more than an experiment, there is a DeprecationWarning that will be raised in Python versions 3.10.9, 3.11.1 and 3.12 and newer about asyncio.get_event_loop(). I believe the correct future proofing moving forward is to use asyncio.new_event_loop(), but I need input from someone more knowledgeable on the subject.

I think the extra CI runners should be removed before merging, as that seems to be a scarce resource(as referenced from the previous PR). A follow-up PR can add a nightly or weekly scheduled run with this active.

Please run multiple times to check for oddities before merging. Afterwards, I can add in the uvloop event loop instead of the default and run that a few more times. However that does require modifying the poetry.lock file.

Pull Request Checklist

Signed-off-by: Jason Little realtyem@gmail.com

sandhose and others added 7 commits October 14, 2022 12:40
@sandhose
Allow enabling the asyncio reactor in complement
8cc7baf
@sandhose
Newsfile.
3ac53ab
@sandhose
Tweak the CI job name
dcf6231
@sandhose
Install the asyncio reactor properly in the complement fork starter
1c71e6d
@realtyem
Merge branch 'quenting-asyncio' into ci-asyncioreactor-take-2
5224f75
@realtyem
Move conditional for SYNAPSE_USE_ASYNCIO_REACTOR to below conditional…
2b4eba2 
… for SYNAPSE_USE_EXPERIMENTAL_FORKING_LAUNCHER.

This allows to use the latter variable to figure out that the asyncio reactor will be used in the forking launcher and
not in __init__.py. As such, rename the variable used in the forking launcher to easily differentiate which import will
be used.
@realtyem
Use the new variable in complement_fork_starter.py
956715d
@realtyem realtyem marked this pull request as ready for review January 16, 2023 22:43
@realtyem realtyem requested a review from a team as a code owner January 16, 2023 22:43
@sandhose sandhose mentioned this pull request Jan 17, 2023
Closed
reivilibre
reivilibre reviewed Jan 17, 2023
View reviewed changes
scripts-dev/complement.sh Outdated Show resolved Hide resolved
.github/workflows/tests.yml Outdated Show resolved Hide resolved
synapse/app/complement_fork_starter.py Show resolved Hide resolved
@realtyem
Copy link
Contributor Author

From my understanding, this would allow properly running async Rust code, via pyo3-asyncio

@realtyem
Address feedback to remove additional CI jobs, remove unnecessary els…
d5016b0 
…e statement. As a drive-by, cleaned up the command line for running the Complement script.
@reivilibre reivilibre requested a review from a team January 18, 2023 11:05
@realtyem
Copy link
Contributor Author

I got 3 test runs(Complement is what to look at):
https://github.com/matrix-org/synapse/actions/runs/3934199323 (two runs on this one)
https://github.com/matrix-org/synapse/actions/runs/3944663353

So it works. Most current test run is after removing the additional CI.

@clokep clokep changed the title Allow enabling the asyncio reactor in complement(take 2 based on 14099) Allow enabling the asyncio reactor in complement Jan 25, 2023
@realtyem realtyem requested a review from squahtx February 1, 2023 22:32
squahtx
squahtx approved these changes Feb 1, 2023
View reviewed changes
Copy link
Contributor

@squahtx squahtx left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Thanks for making the suggested changes!

@squahtx squahtx merged commit 58214db into matrix-org:develop Feb 1, 2023
@realtyem realtyem deleted the ci-asyncioreactor-take-2 branch February 1, 2023 23:58
Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Feb 20, 2023
@Fizzadar
Merge tag 'v1.77.0' into beeper
120c50b 
Synapse 1.77.0 (2023-02-14)
===========================

No significant changes since 1.77.0rc2.

Synapse 1.77.0rc2 (2023-02-10)
==============================

Bugfixes
--------

- Fix bug where retried replication requests would return a failure. Introduced in v1.76.0. ([\matrix-org#15024](matrix-org#15024))

Internal Changes
----------------

- Prepare for future database schema changes. ([\matrix-org#15036](matrix-org#15036))

Synapse 1.77.0rc1 (2023-02-07)
==============================

Features
--------

- Experimental support for [MSC3952](matrix-org/matrix-spec-proposals#3952): intentional mentions. ([\matrix-org#14823](matrix-org#14823), [\matrix-org#14943](matrix-org#14943), [\matrix-org#14957](matrix-org#14957), [\matrix-org#14958](matrix-org#14958))
- Experimental support to suppress notifications from message edits ([MSC3958](matrix-org/matrix-spec-proposals#3958)). ([\matrix-org#14960](matrix-org#14960), [\matrix-org#15016](matrix-org#15016))
- Add profile information, devices and connections to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.77/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\matrix-org#14894](matrix-org#14894))
- Improve performance when joining or sending an event in large rooms. ([\matrix-org#14962](matrix-org#14962))
- Improve performance of joining and leaving large rooms with many local users. ([\matrix-org#14971](matrix-org#14971))

Bugfixes
--------

- Fix a bug introduced in Synapse 1.53.0 where `next_batch` tokens from `/sync` could not be used with the `/relations` endpoint. ([\matrix-org#14866](matrix-org#14866))
- Fix a bug introduced in Synapse 1.35.0 where the module API's `send_local_online_presence_to` would fail to send presence updates over federation. ([\matrix-org#14880](matrix-org#14880))
- Fix a bug introduced in Synapse 1.70.0 where the background updates to add non-thread unique indexes on receipts could fail when upgrading from 1.67.0 or earlier. ([\matrix-org#14915](matrix-org#14915))
- Fix a regression introduced in Synapse 1.69.0 which can result in database corruption when database migrations are interrupted on sqlite. ([\matrix-org#14926](matrix-org#14926))
- Fix a bug introduced in Synapse 1.68.0 where we were unable to service remote joins in rooms with `@room` notification levels set to `null` in their (malformed) power levels. ([\matrix-org#14942](matrix-org#14942))
- Fix a bug introduced in Synapse 1.64.0 where boolean power levels were erroneously permitted in [v10 rooms](https://spec.matrix.org/v1.5/rooms/v10/). ([\matrix-org#14944](matrix-org#14944))
- Fix a long-standing bug where sending messages on servers with presence enabled would spam "Re-starting finished log context" log lines. ([\matrix-org#14947](matrix-org#14947))
- Fix a bug introduced in Synapse 1.68.0 where logging from the Rust module was not properly logged. ([\matrix-org#14976](matrix-org#14976))
- Fix various long-standing bugs in Synapse's config, event and request handling where booleans were unintentionally accepted where an integer was expected. ([\matrix-org#14945](matrix-org#14945))

Internal Changes
----------------

- Add missing type hints. ([\matrix-org#14879](matrix-org#14879), [\matrix-org#14886](matrix-org#14886), [\matrix-org#14887](matrix-org#14887), [\matrix-org#14904](matrix-org#14904), [\matrix-org#14927](matrix-org#14927), [\matrix-org#14956](matrix-org#14956), [\matrix-org#14983](matrix-org#14983), [\matrix-org#14984](matrix-org#14984), [\matrix-org#14985](matrix-org#14985), [\matrix-org#14987](matrix-org#14987), [\matrix-org#14988](matrix-org#14988), [\matrix-org#14990](matrix-org#14990), [\matrix-org#14991](matrix-org#14991), [\matrix-org#14992](matrix-org#14992), [\matrix-org#15007](matrix-org#15007))
- Use `StrCollection` to avoid potential bugs with `Collection[str]`. ([\matrix-org#14922](matrix-org#14922))
- Allow running the complement tests suites with the asyncio reactor enabled. ([\matrix-org#14858](matrix-org#14858))
- Improve performance of `/sync` in a few situations. ([\matrix-org#14908](matrix-org#14908), [\matrix-org#14970](matrix-org#14970))
- Document how to handle Dependabot pull requests. ([\matrix-org#14916](matrix-org#14916))
- Fix typo in release script. ([\matrix-org#14920](matrix-org#14920))
- Update build system requirements to allow building with poetry-core 1.5.0. ([\matrix-org#14949](matrix-org#14949), [\matrix-org#15019](matrix-org#15019))
- Add an [lnav](https://lnav.org) config file for Synapse logs to `/contrib/lnav`. ([\matrix-org#14953](matrix-org#14953))
- Faster joins: Refactor internal handling of servers in room to never store an empty list. ([\matrix-org#14954](matrix-org#14954))
- Faster joins: tag `v2/send_join/` requests to indicate if they served a partial join response. ([\matrix-org#14950](matrix-org#14950))
- Allow running `cargo` without the `extension-module` option. ([\matrix-org#14965](matrix-org#14965))
- Preparatory work for adding a denormalised event stream ordering column in the future. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#14979](matrix-org#14979), [9cd7610](matrix-org@9cd7610), [f10caa7](matrix-org@f10caa7); see [\matrix-org#15014](matrix-org#15014))
- Add tests for `_flatten_dict`. ([\matrix-org#14981](matrix-org#14981), [\matrix-org#15002](matrix-org#15002))

<details><summary>Dependabot updates</summary>

- Bump dtolnay/rust-toolchain from e645b0cf01249a964ec099494d38d2da0f0b349f to 9cd00a88a73addc8617065438eff914dd08d0955. ([\matrix-org#14968](matrix-org#14968))
- Bump docker/build-push-action from 3 to 4. ([\matrix-org#14952](matrix-org#14952))
- Bump ijson from 3.1.4 to 3.2.0.post0. ([\matrix-org#14935](matrix-org#14935))
- Bump types-pyyaml from 6.0.12.2 to 6.0.12.3. ([\matrix-org#14936](matrix-org#14936))
- Bump types-jsonschema from 4.17.0.2 to 4.17.0.3. ([\matrix-org#14937](matrix-org#14937))
- Bump types-pillow from 9.4.0.3 to 9.4.0.5. ([\matrix-org#14938](matrix-org#14938))
- Bump hiredis from 2.0.0 to 2.1.1. ([\matrix-org#14939](matrix-org#14939))
- Bump hiredis from 2.1.1 to 2.2.1. ([\matrix-org#14993](matrix-org#14993))
- Bump types-setuptools from 65.6.0.3 to 67.1.0.0. ([\matrix-org#14994](matrix-org#14994))
- Bump prometheus-client from 0.15.0 to 0.16.0. ([\matrix-org#14995](matrix-org#14995))
- Bump anyhow from 1.0.68 to 1.0.69. ([\matrix-org#14996](matrix-org#14996))
- Bump serde_json from 1.0.91 to 1.0.92. ([\matrix-org#14997](matrix-org#14997))
- Bump isort from 5.11.4 to 5.11.5. ([\matrix-org#14998](matrix-org#14998))
- Bump phonenumbers from 8.13.4 to 8.13.5. ([\matrix-org#14999](matrix-org#14999))
</details>

# -----BEGIN PGP SIGNATURE-----
#
# iHUEABYKAB0WIQSTI7xPaHQ1yo0PA8uSL1esuTqr+QUCY+ubcgAKCRCSL1esuTqr
# +foKAP9K8HQeGlOns6GRRiyY1EPILRvptAXeMit2eQ19J+ROKAD+JZM5WqlpWAdW
# ikmC4GV8hps01IAWFwKtK3+pLqg79gc=
# =yBT7
# -----END PGP SIGNATURE-----
# gpg: Signature made Tue Feb 14 14:32:18 2023 GMT
# gpg:                using EDDSA key 9323BC4F687435CA8D0F03CB922F57ACB93AABF9
# gpg: Can't check signature: No public key

# Conflicts:
#	docker/Dockerfile
#	poetry.lock
#	rust/src/push/base_rules.rs
#	rust/src/push/evaluator.rs
#	rust/src/push/mod.rs
#	synapse/config/experimental.py
#	synapse/event_auth.py
#	synapse/handlers/message.py
#	synapse/handlers/pagination.py
#	synapse/push/bulk_push_rule_evaluator.py
#	synapse/rest/admin/rooms.py
#	synapse/storage/databases/main/devices.py
#	synapse/storage/databases/main/roommember.py
#	tests/push/test_push_rule_evaluator.py
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Mar 1, 2023
@Midar
Update chat/matrix-synapse to 1.77.0
1cf80f5 
Synapse 1.77.0 (2023-02-14)
===========================

No significant changes since 1.77.0rc2.


Synapse 1.77.0rc2 (2023-02-10)
==============================

Bugfixes
--------

- Fix bug where retried replication requests would return a failure. Introduced in v1.76.0. ([\#15024](matrix-org/synapse#15024))


Internal Changes
----------------

- Prepare for future database schema changes. ([\#15036](matrix-org/synapse#15036))


Synapse 1.77.0rc1 (2023-02-07)
==============================

Features
--------

- Experimental support for [MSC3952](matrix-org/matrix-spec-proposals#3952): intentional mentions. ([\#14823](matrix-org/synapse#14823), [\#14943](matrix-org/synapse#14943), [\#14957](matrix-org/synapse#14957), [\#14958](matrix-org/synapse#14958))
- Experimental support to suppress notifications from message edits ([MSC3958](matrix-org/matrix-spec-proposals#3958)). ([\#14960](matrix-org/synapse#14960), [\#15016](matrix-org/synapse#15016))
- Add profile information, devices and connections to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.77/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\#14894](matrix-org/synapse#14894))
- Improve performance when joining or sending an event in large rooms. ([\#14962](matrix-org/synapse#14962))
- Improve performance of joining and leaving large rooms with many local users. ([\#14971](matrix-org/synapse#14971))


Bugfixes
--------

- Fix a bug introduced in Synapse 1.53.0 where `next_batch` tokens from `/sync` could not be used with the `/relations` endpoint. ([\#14866](matrix-org/synapse#14866))
- Fix a bug introduced in Synapse 1.35.0 where the module API's `send_local_online_presence_to` would fail to send presence updates over federation. ([\#14880](matrix-org/synapse#14880))
- Fix a bug introduced in Synapse 1.70.0 where the background updates to add non-thread unique indexes on receipts could fail when upgrading from 1.67.0 or earlier. ([\#14915](matrix-org/synapse#14915))
- Fix a regression introduced in Synapse 1.69.0 which can result in database corruption when database migrations are interrupted on sqlite. ([\#14926](matrix-org/synapse#14926))
- Fix a bug introduced in Synapse 1.68.0 where we were unable to service remote joins in rooms with `@room` notification levels set to `null` in their (malformed) power levels. ([\#14942](matrix-org/synapse#14942))
- Fix a bug introduced in Synapse 1.64.0 where boolean power levels were erroneously permitted in [v10 rooms](https://spec.matrix.org/v1.5/rooms/v10/). ([\#14944](matrix-org/synapse#14944))
- Fix a long-standing bug where sending messages on servers with presence enabled would spam "Re-starting finished log context" log lines. ([\#14947](matrix-org/synapse#14947))
- Fix a bug introduced in Synapse 1.68.0 where logging from the Rust module was not properly logged. ([\#14976](matrix-org/synapse#14976))
- Fix various long-standing bugs in Synapse's config, event and request handling where booleans were unintentionally accepted where an integer was expected. ([\#14945](matrix-org/synapse#14945))


Internal Changes
----------------

- Add missing type hints. ([\#14879](matrix-org/synapse#14879), [\#14886](matrix-org/synapse#14886), [\#14887](matrix-org/synapse#14887), [\#14904](matrix-org/synapse#14904), [\#14927](matrix-org/synapse#14927), [\#14956](matrix-org/synapse#14956), [\#14983](matrix-org/synapse#14983), [\#14984](matrix-org/synapse#14984), [\#14985](matrix-org/synapse#14985), [\#14987](matrix-org/synapse#14987), [\#14988](matrix-org/synapse#14988), [\#14990](matrix-org/synapse#14990), [\#14991](matrix-org/synapse#14991), [\#14992](matrix-org/synapse#14992), [\#15007](matrix-org/synapse#15007))
- Use `StrCollection` to avoid potential bugs with `Collection[str]`. ([\#14922](matrix-org/synapse#14922))
- Allow running the complement tests suites with the asyncio reactor enabled. ([\#14858](matrix-org/synapse#14858))
- Improve performance of `/sync` in a few situations. ([\#14908](matrix-org/synapse#14908), [\#14970](matrix-org/synapse#14970))
- Document how to handle Dependabot pull requests. ([\#14916](matrix-org/synapse#14916))
- Fix typo in release script. ([\#14920](matrix-org/synapse#14920))
- Update build system requirements to allow building with poetry-core 1.5.0. ([\#14949](matrix-org/synapse#14949), [\#15019](matrix-org/synapse#15019))
- Add an [lnav](https://lnav.org) config file for Synapse logs to `/contrib/lnav`. ([\#14953](matrix-org/synapse#14953))
- Faster joins: Refactor internal handling of servers in room to never store an empty list. ([\#14954](matrix-org/synapse#14954))
- Faster joins: tag `v2/send_join/` requests to indicate if they served a partial join response. ([\#14950](matrix-org/synapse#14950))
- Allow running `cargo` without the `extension-module` option. ([\#14965](matrix-org/synapse#14965))
- Preparatory work for adding a denormalised event stream ordering column in the future. Contributed by Nick @ Beeper (@Fizzadar). ([\#14979](matrix-org/synapse#14979), [9cd7610](matrix-org/synapse@9cd7610), [f10caa7](matrix-org/synapse@f10caa7); see [\#15014](matrix-org/synapse#15014))
- Add tests for `_flatten_dict`. ([\#14981](matrix-org/synapse#14981), [\#15002](matrix-org/synapse#15002))

<details><summary>Locked dependency updates</summary>

- Bump dtolnay/rust-toolchain from e645b0cf01249a964ec099494d38d2da0f0b349f to 9cd00a88a73addc8617065438eff914dd08d0955. ([\#14968](matrix-org/synapse#14968))
- Bump docker/build-push-action from 3 to 4. ([\#14952](matrix-org/synapse#14952))
- Bump ijson from 3.1.4 to 3.2.0.post0. ([\#14935](matrix-org/synapse#14935))
- Bump types-pyyaml from 6.0.12.2 to 6.0.12.3. ([\#14936](matrix-org/synapse#14936))
- Bump types-jsonschema from 4.17.0.2 to 4.17.0.3. ([\#14937](matrix-org/synapse#14937))
- Bump types-pillow from 9.4.0.3 to 9.4.0.5. ([\#14938](matrix-org/synapse#14938))
- Bump hiredis from 2.0.0 to 2.1.1. ([\#14939](matrix-org/synapse#14939))
- Bump hiredis from 2.1.1 to 2.2.1. ([\#14993](matrix-org/synapse#14993))
- Bump types-setuptools from 65.6.0.3 to 67.1.0.0. ([\#14994](matrix-org/synapse#14994))
- Bump prometheus-client from 0.15.0 to 0.16.0. ([\#14995](matrix-org/synapse#14995))
- Bump anyhow from 1.0.68 to 1.0.69. ([\#14996](matrix-org/synapse#14996))
- Bump serde_json from 1.0.91 to 1.0.92. ([\#14997](matrix-org/synapse#14997))
- Bump isort from 5.11.4 to 5.11.5. ([\#14998](matrix-org/synapse#14998))
- Bump phonenumbers from 8.13.4 to 8.13.5. ([\#14999](matrix-org/synapse#14999))
</details>
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Reviewers

@reivilibre reivilibre reivilibre left review comments

@squahtx squahtx squahtx approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
4 participants
@realtyem @squahtx @reivilibre @sandhose