Allow background tasks to be run on a separate worker. #8369

clokep · 2020-09-21T18:16:52Z

The overall plan here is to run (some) background tasks on a separate worker, instead of the main process. This causes two different changes:

Some tasks will now be run off the main worker.
Some tasks will now only be run by a single worker. (This was inconsistent previously, some were only run on the main worker while others were run on all workers.)

I'm hoping this will have the following benefits:

Not having background tasks potentially slow-down user facing requests.
Not running the same task multiple times (which likely end up trying to modify the same database rows, causing contention).
From some rough math (see Profile the matrix.org master process #7670) I think this will eventually allow moving 10 - 15% (or maybe even more) CPU load off the main process. (Although this PR only moves a few tasks.)

I mostly found these be searching for callers of looping_call. Note that some of the callers use in process (in-memory) state, while others use database state. Only the latter is applicable to be moved to a separate process.

I simplified this to only move a few tasks to the new worker. I think this is clearer than trying to move them all at once. Follow-up PRs can move additional tasks.

By default the main process still runs background tasks.

Related to #8500.

synapse/app/homeserver.py

clokep · 2020-09-22T15:05:14Z

I would like to do a bit more testing locally, but I think this is in a place where some feedback is warranted!

erikjohnston

Otherwise looks OK (though haven't looked in depth)

synapse/app/generic_worker.py

synapse/app/homeserver.py

synapse/app/generic_worker.py

clokep · 2020-09-23T12:01:35Z

Running this manually I've run into two issues:

A worker with no listeners doesn't end up importing the handlers to actually register the background tasks.
We need to confirm that the workers have access to the proper methods on the store (e.g. delete_old_ui_auth_sessions is only available to the main process).

@erikjohnston Any thoughts on approaches for the above?

erikjohnston · 2020-09-23T13:28:41Z

Running this manually I've run into two issues:

A worker with no listeners doesn't end up importing the handlers to actually register the background tasks.

We need to confirm that the workers have access to the proper methods on the store (e.g. delete_old_ui_auth_sessions is only available to the main process).

@erikjohnston Any thoughts on approaches for the above?

Hmm, I don't really. We do manually call hs.get_foo() on startup occasionally to ensure some things are built, but that doesn't sound like the right thing to do for each place we do start a background job :/

clokep · 2020-09-24T14:47:58Z

Another thought that came up is that we can setup the listener for this, but not actually route anything to it. This wouldn't be ideal, but means we don't need to refactor a bunch of stuff.

clokep · 2020-09-24T19:22:34Z

Running this manually I've run into two issues:

A worker with no listeners doesn't end up importing the handlers to actually register the background tasks.

We need to confirm that the workers have access to the proper methods on the store (e.g. delete_old_ui_auth_sessions is only available to the main process).

@erikjohnston Any thoughts on approaches for the above?

Hmm, I don't really. We do manually call hs.get_foo() on startup occasionally to ensure some things are built, but that doesn't sound like the right thing to do for each place we do start a background job :/

Turns out we already have a mechanism to do this... 🤦 I extended it a bit to include the extra handlers and call it for the background worker. See 2e98b78 for this approach. It somewhat feels like a thing we need to remember in the future, but not sure there's a better way.

We need to confirm that the workers have access to the proper methods on the store (e.g. delete_old_ui_auth_sessions is only available to the main process).

This is actually causing me more pain at the moment as it will require moving quite a few methods. I'm not sure how safe this is, but I'll give it a go.

synapse/config/workers.py

clokep · 2020-09-25T12:48:21Z

I believe I've solved the current two issues although I'm still testing this. Please take another look! 👁️

clokep · 2020-09-25T13:59:27Z

I did a bit of testing with this and I'm fairly certain it works.

erikjohnston · 2020-09-29T09:38:13Z

Yeah, I think this looks sane

clokep · 2020-09-29T14:00:47Z

synapse/storage/databases/main/ui_auth.py

+class UIAuthStore(UIAuthWorkerStore):
+    pass


Would it make more sense to remove the UIAuthWorkerStore and just use UIAuthStore everywhere instead?

Potentially, though its not the worse to keep the separation for consistency.

clokep · 2020-09-29T14:03:46Z

@erikjohnston I managed to fix the sytests, I think this and matrix-org/sytest#958 are reviewable now! Thanks for your help!

We talked about it in #synapse-dev:matrix.org, but to re-iterate here: this backs out the changes to the user directory background task since that already had a separate flag (update_user_directory) which is used if you split out the user directory endpoint.

synapse/app/homeserver.py

erikjohnston · 2020-09-30T09:27:53Z

synapse/storage/databases/main/ui_auth.py

+class UIAuthStore(UIAuthWorkerStore):
+    pass


Potentially, though its not the worse to keep the separation for consistency.

synapse/config/workers.py

erikjohnston · 2020-09-30T14:53:48Z

synapse/app/generic_worker.py

+    # If background tasks are running on this worker, start collecting the phone
+    # home stats.
+    if hs.config.run_background_tasks:
+        start_phone_stats_home(hs)


I wonder if we can move this to somewhere in synapse.app._base?

Perhaps around

synapse/synapse/app/_base.py

Lines 274 to 275 in c619253

setup_sentry(hs)

setup_sdnotify(hs)

would make sense?

The only downside of this is that it also gets used for synapse.app.admin_cmd. I don't know how frequently this gets run and if it would cause any issues. Thoughts @erikjohnston?

Wouldn't the admin command have the run_background_tasks disabled?

Ah, there's a section to disable all background tasks anyway, so looks like I should add that there.

…vailable to workers.

…task-worker

clokep · 2020-10-01T14:22:28Z

@erikjohnston Could you give this another quick look to make sure I didn't do anything silly with moving stuff between data stores?

erikjohnston

I think this still looks sane!

…rd_party_event_rules_public_rooms * 'develop' of github.com:matrix-org/synapse: (109 commits) Add logging on startup/shutdown (#8448) Speed up unit tests when using PostgreSQL (#8450) Allow background tasks to be run on a separate worker. (#8369) move #8444 to 'feature' linkify changelog 1.21.0rc2 1.21.0rc2 Fix bug in remote thumbnail search (#8438) Include a public_baseurl in configs generated by the demo script. (#8443) Fix DB query on startup for negative streams. (#8447) Convert additional templates to Jinja (#8444) Fix malformed log line in new federation "catch up" logic (#8442) Add unit test for event persister sharding (#8433) Add config option for always using "userinfo endpoint" for OIDC (#7658) Do not expose the experimental appservice login flow to clients. (#8440) update changelog fix a logging error in thumbnailer (#8435) changelog fixes fix version number 1.21.0 ...

@Akkowicz

Synapse 1.22.0rc1 (2020-10-22) ============================== Features -------- - Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](#7658)) - Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](#8292), [\#8467](#8467)) - Add support for olm fallback keys ([MSC2732](matrix-org/matrix-spec-proposals#2732)). ([\#8312](#8312), [\#8501](#8501)) - Add support for running background tasks in a separate worker process. ([\#8369](#8369), [\#8458](#8458), [\#8489](#8489), [\#8513](#8513), [\#8544](#8544), [\#8599](#8599)) - Add support for device dehydration ([MSC2697](matrix-org/matrix-spec-proposals#2697)). ([\#8380](#8380)) - Add support for [MSC2409](matrix-org/matrix-spec-proposals#2409), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](#8437), [\#8590](#8590)) - Change default room version to "6", per [MSC2788](matrix-org/matrix-spec-proposals#2788). ([\#8461](#8461)) - Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](#8479)) - Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](#8502)) - Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](#8535), [\#8564](#8564)) Bugfixes -------- - Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](#8454)) - Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](#8457)) - Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](#8465)) - Fix message duplication if something goes wrong after persisting the event. ([\#8476](#8476)) - Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](#8486)) - Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](#8504)) - Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](#8517)) - Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](#8527)) - Fix not sending events over federation when using sharded event writers. ([\#8536](#8536)) - Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](#8545)) - Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](#8567)) - Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](#8585)) - Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](#8606)) Updates to the Docker image --------------------------- - Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](#7921)) - Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](#8390)) Improved Documentation ---------------------- - Update the directions for using the manhole with coroutines. ([\#8462](#8462)) - Improve readme by adding new shield.io badges. ([\#8493](#8493)) - Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @maquis196. ([\#8526](#8526)) - Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](#8529)) Deprecations and Removals ------------------------- - Drop unused `device_max_stream_id` table. ([\#8589](#8589)) Internal Changes ---------------- - Check for unreachable code with mypy. ([\#8432](#8432)) - Add unit test for event persister sharding. ([\#8433](#8433)) - Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](#8439), [\#8488](#8488), [\#8496](#8496), [\#8499](#8499)) - Configure `public_baseurl` when using demo scripts. ([\#8443](#8443)) - Add SQL logging on queries that happen during startup. ([\#8448](#8448)) - Speed up unit tests when using PostgreSQL. ([\#8450](#8450)) - Remove redundant database loads of stream_ordering for events we already have. ([\#8452](#8452)) - Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](#8463)) - Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](#8464)) - Additional testing for `ThirdPartyEventRules`. ([\#8468](#8468)) - Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](#8472)) - Unblacklist some sytests. ([\#8474](#8474)) - Include the log level in the phone home stats. ([\#8477](#8477)) - Remove outdated sphinx documentation, scripts and configuration. ([\#8480](#8480)) - Clarify error message when plugin config parsers raise an error. ([\#8492](#8492)) - Remove the deprecated `Handlers` object. ([\#8494](#8494)) - Fix a threadsafety bug in unit tests. ([\#8497](#8497)) - Add user agent to user_daily_visits table. ([\#8503](#8503)) - Add type hints to various parts of the code base. ([\#8407](#8407), [\#8505](#8505), [\#8507](#8507), [\#8547](#8547), [\#8562](#8562), [\#8609](#8609)) - Remove unused code from the test framework. ([\#8514](#8514)) - Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](#8515)) - Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](#8537)) - Improve database performance by executing more queries without starting transactions. ([\#8542](#8542)) - Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](#8548)) - Move metric registration code down into `LruCache`. ([\#8561](#8561), [\#8591](#8591)) - Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](#8563)) - Add virtualenv-generated folders to `.gitignore`. ([\#8566](#8566)) - Add `get_immediate` method to `DeferredCache`. ([\#8568](#8568)) - Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](#8569)) - Fix `synmark` benchmark runner. ([\#8571](#8571)) - Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](#8572)) - Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](#8577)) - Support macOS on the `synmark` benchmark runner. ([\#8578](#8578)) - Update `mypy` static type checker to 0.790. ([\#8583](#8583), [\#8600](#8600)) - Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](#8587)) - Remove extraneous unittest logging decorators from unit tests. ([\#8592](#8592)) - Minor optimisations in caching code. ([\#8593](#8593), [\#8594](#8594))

@Akkowicz

Synapse 1.22.0 (2020-10-27) =========================== No significant changes. Synapse 1.22.0rc2 (2020-10-26) ============================== Bugfixes -------- - Fix bugs where ephemeral events were not sent to appservices. Broke in v1.22.0rc1. ([\#8648](matrix-org/synapse#8648), [\#8656](matrix-org/synapse#8656)) - Fix `user_daily_visits` table to not have duplicate rows per user/device due to multiple user agents. Broke in v1.22.0rc1. ([\#8654](matrix-org/synapse#8654)) Synapse 1.22.0rc1 (2020-10-22) ============================== Features -------- - Add a configuration option for always using the "userinfo endpoint" for OpenID Connect. This fixes support for some identity providers, e.g. GitLab. Contributed by Benjamin Koch. ([\#7658](matrix-org/synapse#7658)) - Add ability for `ThirdPartyEventRules` modules to query and manipulate whether a room is in the public rooms directory. ([\#8292](matrix-org/synapse#8292), [\#8467](matrix-org/synapse#8467)) - Add support for olm fallback keys ([MSC2732](matrix-org/matrix-spec-proposals#2732)). ([\#8312](matrix-org/synapse#8312), [\#8501](matrix-org/synapse#8501)) - Add support for running background tasks in a separate worker process. ([\#8369](matrix-org/synapse#8369), [\#8458](matrix-org/synapse#8458), [\#8489](matrix-org/synapse#8489), [\#8513](matrix-org/synapse#8513), [\#8544](matrix-org/synapse#8544), [\#8599](matrix-org/synapse#8599)) - Add support for device dehydration ([MSC2697](matrix-org/matrix-spec-proposals#2697)). ([\#8380](matrix-org/synapse#8380)) - Add support for [MSC2409](matrix-org/matrix-spec-proposals#2409), which allows sending typing, read receipts, and presence events to appservices. ([\#8437](matrix-org/synapse#8437), [\#8590](matrix-org/synapse#8590)) - Change default room version to "6", per [MSC2788](matrix-org/matrix-spec-proposals#2788). ([\#8461](matrix-org/synapse#8461)) - Add the ability to send non-membership events into a room via the `ModuleApi`. ([\#8479](matrix-org/synapse#8479)) - Increase default upload size limit from 10M to 50M. Contributed by @Akkowicz. ([\#8502](matrix-org/synapse#8502)) - Add support for modifying event content in `ThirdPartyRules` modules. ([\#8535](matrix-org/synapse#8535), [\#8564](matrix-org/synapse#8564)) Bugfixes -------- - Fix a longstanding bug where invalid ignored users in account data could break clients. ([\#8454](matrix-org/synapse#8454)) - Fix a bug where backfilling a room with an event that was missing the `redacts` field would break. ([\#8457](matrix-org/synapse#8457)) - Don't attempt to respond to some requests if the client has already disconnected. ([\#8465](matrix-org/synapse#8465)) - Fix message duplication if something goes wrong after persisting the event. ([\#8476](matrix-org/synapse#8476)) - Fix incremental sync returning an incorrect `prev_batch` token in timeline section, which when used to paginate returned events that were included in the incremental sync. Broken since v0.16.0. ([\#8486](matrix-org/synapse#8486)) - Expose the `uk.half-shot.msc2778.login.application_service` to clients from the login API. This feature was added in v1.21.0, but was not exposed as a potential login flow. ([\#8504](matrix-org/synapse#8504)) - Fix error code for `/profile/{userId}/displayname` to be `M_BAD_JSON`. ([\#8517](matrix-org/synapse#8517)) - Fix a bug introduced in v1.7.0 that could cause Synapse to insert values from non-state `m.room.retention` events into the `room_retention` database table. ([\#8527](matrix-org/synapse#8527)) - Fix not sending events over federation when using sharded event writers. ([\#8536](matrix-org/synapse#8536)) - Fix a long standing bug where email notifications for encrypted messages were blank. ([\#8545](matrix-org/synapse#8545)) - Fix increase in the number of `There was no active span...` errors logged when using OpenTracing. ([\#8567](matrix-org/synapse#8567)) - Fix a bug that prevented errors encountered during execution of the `synapse_port_db` from being correctly printed. ([\#8585](matrix-org/synapse#8585)) - Fix appservice transactions to only include a maximum of 100 persistent and 100 ephemeral events. ([\#8606](matrix-org/synapse#8606)) Updates to the Docker image --------------------------- - Added multi-arch support (arm64,arm/v7) for the docker images. Contributed by @maquis196. ([\#7921](matrix-org/synapse#7921)) - Add support for passing commandline args to the synapse process. Contributed by @samuel-p. ([\#8390](matrix-org/synapse#8390)) Improved Documentation ---------------------- - Update the directions for using the manhole with coroutines. ([\#8462](matrix-org/synapse#8462)) - Improve readme by adding new shield.io badges. ([\#8493](matrix-org/synapse#8493)) - Added note about docker in manhole.md regarding which ip address to bind to. Contributed by @maquis196. ([\#8526](matrix-org/synapse#8526)) - Document the new behaviour of the `allowed_lifetime_min` and `allowed_lifetime_max` settings in the room retention configuration. ([\#8529](matrix-org/synapse#8529)) Deprecations and Removals ------------------------- - Drop unused `device_max_stream_id` table. ([\#8589](matrix-org/synapse#8589)) Internal Changes ---------------- - Check for unreachable code with mypy. ([\#8432](matrix-org/synapse#8432)) - Add unit test for event persister sharding. ([\#8433](matrix-org/synapse#8433)) - Allow events to be sent to clients sooner when using sharded event persisters. ([\#8439](matrix-org/synapse#8439), [\#8488](matrix-org/synapse#8488), [\#8496](matrix-org/synapse#8496), [\#8499](matrix-org/synapse#8499)) - Configure `public_baseurl` when using demo scripts. ([\#8443](matrix-org/synapse#8443)) - Add SQL logging on queries that happen during startup. ([\#8448](matrix-org/synapse#8448)) - Speed up unit tests when using PostgreSQL. ([\#8450](matrix-org/synapse#8450)) - Remove redundant database loads of stream_ordering for events we already have. ([\#8452](matrix-org/synapse#8452)) - Reduce inconsistencies between codepaths for membership and non-membership events. ([\#8463](matrix-org/synapse#8463)) - Combine `SpamCheckerApi` with the more generic `ModuleApi`. ([\#8464](matrix-org/synapse#8464)) - Additional testing for `ThirdPartyEventRules`. ([\#8468](matrix-org/synapse#8468)) - Add `-d` option to `./scripts-dev/lint.sh` to lint files that have changed since the last git commit. ([\#8472](matrix-org/synapse#8472)) - Unblacklist some sytests. ([\#8474](matrix-org/synapse#8474)) - Include the log level in the phone home stats. ([\#8477](matrix-org/synapse#8477)) - Remove outdated sphinx documentation, scripts and configuration. ([\#8480](matrix-org/synapse#8480)) - Clarify error message when plugin config parsers raise an error. ([\#8492](matrix-org/synapse#8492)) - Remove the deprecated `Handlers` object. ([\#8494](matrix-org/synapse#8494)) - Fix a threadsafety bug in unit tests. ([\#8497](matrix-org/synapse#8497)) - Add user agent to user_daily_visits table. ([\#8503](matrix-org/synapse#8503)) - Add type hints to various parts of the code base. ([\#8407](matrix-org/synapse#8407), [\#8505](matrix-org/synapse#8505), [\#8507](matrix-org/synapse#8507), [\#8547](matrix-org/synapse#8547), [\#8562](matrix-org/synapse#8562), [\#8609](matrix-org/synapse#8609)) - Remove unused code from the test framework. ([\#8514](matrix-org/synapse#8514)) - Apply some internal fixes to the `HomeServer` class to make its code more idiomatic and statically-verifiable. ([\#8515](matrix-org/synapse#8515)) - Factor out common code between `RoomMemberHandler._locally_reject_invite` and `EventCreationHandler.create_event`. ([\#8537](matrix-org/synapse#8537)) - Improve database performance by executing more queries without starting transactions. ([\#8542](matrix-org/synapse#8542)) - Rename `Cache` to `DeferredCache`, to better reflect its purpose. ([\#8548](matrix-org/synapse#8548)) - Move metric registration code down into `LruCache`. ([\#8561](matrix-org/synapse#8561), [\#8591](matrix-org/synapse#8591)) - Replace `DeferredCache` with the lighter-weight `LruCache` where possible. ([\#8563](matrix-org/synapse#8563)) - Add virtualenv-generated folders to `.gitignore`. ([\#8566](matrix-org/synapse#8566)) - Add `get_immediate` method to `DeferredCache`. ([\#8568](matrix-org/synapse#8568)) - Fix mypy not properly checking across the codebase, additionally, fix a typing assertion error in `handlers/auth.py`. ([\#8569](matrix-org/synapse#8569)) - Fix `synmark` benchmark runner. ([\#8571](matrix-org/synapse#8571)) - Modify `DeferredCache.get()` to return `Deferred`s instead of `ObservableDeferred`s. ([\#8572](matrix-org/synapse#8572)) - Adjust a protocol-type definition to fit `sqlite3` assertions. ([\#8577](matrix-org/synapse#8577)) - Support macOS on the `synmark` benchmark runner. ([\#8578](matrix-org/synapse#8578)) - Update `mypy` static type checker to 0.790. ([\#8583](matrix-org/synapse#8583), [\#8600](matrix-org/synapse#8600)) - Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8587](matrix-org/synapse#8587)) - Remove extraneous unittest logging decorators from unit tests. ([\#8592](matrix-org/synapse#8592)) - Minor optimisations in caching code. ([\#8593](matrix-org/synapse#8593), [\#8594](matrix-org/synapse#8594))

Allow background tasks to be run on a separate worker.

71ade09

clokep force-pushed the clokep/background-task-worker branch from d690d9c to 71ade09 Compare September 21, 2020 19:05

clokep mentioned this pull request Sep 22, 2020

Run the background worker during worker mode. matrix-org/sytest#958

Merged

clokep commented Sep 22, 2020

View reviewed changes

synapse/app/homeserver.py Outdated Show resolved Hide resolved

clokep commented Sep 22, 2020

View reviewed changes

synapse/app/homeserver.py Outdated Show resolved Hide resolved

clokep requested a review from a team September 22, 2020 15:02

erikjohnston reviewed Sep 22, 2020

View reviewed changes

synapse/app/generic_worker.py Outdated Show resolved Hide resolved

synapse/app/homeserver.py Outdated Show resolved Hide resolved

synapse/app/homeserver.py Outdated Show resolved Hide resolved

clokep added 3 commits September 22, 2020 13:27

Only run function on background worker.

094b11e

Accept a worker name instead of using a flag.

fd8aad3

Remove check for being the background_worker app.

67c6fa4

clokep commented Sep 22, 2020

View reviewed changes

synapse/app/generic_worker.py Outdated Show resolved Hide resolved

Do not allow a non-existent worker app.

08a7d5d

clokep added 2 commits September 24, 2020 14:25

Also run the user directory updating in the backgronud.

0d06a87

Ensure the proper handlers are loaded during start-up.

2e98b78

clokep added 5 commits September 25, 2020 07:29

Backout some of the changes to simplify the PR.

b77b89b

Ensure that the background tasks have access to the proper stores.

f26b92e

Add some documentation.

53a4402

Do not require a replication endpoint for the instance map.

82d167b

Clarify confusing wording.

0c9e970

clokep commented Sep 25, 2020

View reviewed changes

synapse/config/workers.py Outdated Show resolved Hide resolved

clokep requested a review from erikjohnston September 25, 2020 12:47

clokep self-assigned this Sep 28, 2020

clokep added 2 commits September 29, 2020 07:47

Do not require the background worker instance to be in the instance map.

d40aff7

Update the sample config.

c015379

clokep force-pushed the clokep/background-task-worker branch from 9a2c42f to c015379 Compare September 29, 2020 11:49

The user directory background tasks are controlled by a separate flag.

cfe28f2

clokep commented Sep 29, 2020

View reviewed changes

clokep marked this pull request as ready for review September 29, 2020 14:01

erikjohnston suggested changes Sep 30, 2020

View reviewed changes

clokep added 2 commits September 30, 2020 08:04

Rename instance variable.

f0c83d9

Allow the phone home stats to be able to run on any worker.

b226c49

erikjohnston reviewed Sep 30, 2020

View reviewed changes

erikjohnston approved these changes Sep 30, 2020

View reviewed changes

clokep added 3 commits September 30, 2020 11:13

Move around some storage classes to make the proper metrics methods a…

cb740cf

…vailable to workers.

Consolidate logic for starting metrics calls.

179d8c3

Merge remote-tracking branch 'origin/develop' into clokep/background-…

9fb2c05

…task-worker

clokep requested a review from erikjohnston October 1, 2020 14:22

erikjohnston approved these changes Oct 1, 2020

View reviewed changes

clokep merged commit 6289467 into develop Oct 2, 2020

clokep deleted the clokep/background-task-worker branch October 2, 2020 12:23

clokep mentioned this pull request Oct 2, 2020

Move additional tasks to the background worker #8458

Merged

clokep added a commit that referenced this pull request Oct 7, 2020

Apply wrap_as_background_process to changes from #8369.

74a12fb

clokep mentioned this pull request Oct 7, 2020

Move additional tasks to the background worker, part 3 #8489

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Allow background tasks to be run on a separate worker. #8369

Allow background tasks to be run on a separate worker. #8369

clokep commented Sep 21, 2020 •

edited

Loading

clokep commented Sep 22, 2020

erikjohnston left a comment

clokep commented Sep 23, 2020

erikjohnston commented Sep 23, 2020

clokep commented Sep 24, 2020

clokep commented Sep 24, 2020

clokep commented Sep 25, 2020

clokep commented Sep 25, 2020

erikjohnston commented Sep 29, 2020

clokep Sep 29, 2020

erikjohnston Sep 30, 2020

clokep commented Sep 29, 2020

erikjohnston Sep 30, 2020

erikjohnston Sep 30, 2020

clokep Sep 30, 2020

clokep Sep 30, 2020

erikjohnston Sep 30, 2020

clokep Sep 30, 2020

clokep commented Oct 1, 2020

erikjohnston left a comment

Allow background tasks to be run on a separate worker. #8369

Allow background tasks to be run on a separate worker. #8369

Conversation

clokep commented Sep 21, 2020 • edited Loading

clokep commented Sep 22, 2020

erikjohnston left a comment

Choose a reason for hiding this comment

clokep commented Sep 23, 2020

erikjohnston commented Sep 23, 2020

clokep commented Sep 24, 2020

clokep commented Sep 24, 2020

clokep commented Sep 25, 2020

clokep commented Sep 25, 2020

erikjohnston commented Sep 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep commented Sep 29, 2020

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

clokep commented Oct 1, 2020

erikjohnston left a comment

Choose a reason for hiding this comment

clokep commented Sep 21, 2020 •

edited

Loading