Shard master process.

[erikjohnston]

Follows on from: #6677

Now that Synapse supports writing events from a different process, we should look at supporting writing events from multiple processes at once, aka sharding of events stream.

[djschilling]

Is there a documentation how this can be achieved?

[erikjohnston]

https://docs.google.com/document/d/1xL1foUsHgcHdVTOAMSJIST04cH0ukHd-oCKFw3m9sxY has our current thoughts, we've already done some of that such as sharding the cache invalidation stream.

[djschilling]

What i ment was the documentation of what is currently possible with Now that Synapse supports writing events from a different process

[erikjohnston]

Ah, right. There isn't any yet, but we're aiming to get some docs for the next release. Right now its quite experimental and hasn't been tested a huge amount, so we don't recommend anyone try it out on production quite yet (unless they really really know what they're doing)

[benlangfeld]

Ah, right. There isn't any yet, but we're aiming to get some docs for the next release. Right now its quite experimental and hasn't been tested a huge amount, so we don't recommend anyone try it out on production quite yet (unless they really really know what they're doing)

Hey @erikjohnston. Did that documentation ever get written?

[erikjohnston]

Ah, right. There isn't any yet, but we're aiming to get some docs for the next release. Right now its quite experimental and hasn't been tested a huge amount, so we don't recommend anyone try it out on production quite yet (unless they really really know what they're doing)

Hey @erikjohnston. Did that documentation ever get written?

Yup, how to move event persistence off master is now at: https://github.com/matrix-org/synapse/blob/develop/docs/workers.md#stream-writers

[benlangfeld]

Ah, right. There isn't any yet, but we're aiming to get some docs for the next release. Right now its quite experimental and hasn't been tested a huge amount, so we don't recommend anyone try it out on production quite yet (unless they really really know what they're doing)

Hey @erikjohnston. Did that documentation ever get written?

Yup, how to move event persistence off master is now at: https://github.com/matrix-org/synapse/blob/develop/docs/workers.md#stream-writers

Does that mean that this issue is complete? If not, what remains to be done?

Is there any documentation which clarifies why the master process still exists, and why it must be a singleton?

[benlangfeld]

Hey @erikjohnston , I wonder if maybe having skipped mentioning you meant you didn't see that question. I appreciate your time in helping understand this. We're evaluating Synapse for a use case that might justify our contributing or funding the contribution of improvements to HA.

[erikjohnston]

I'm afraid there isn't an authoritative list of what still remains on master, broadly we've been moving stuff of master whenever we start seeing it become a problem, but that does mean there are quite a few APIs and streams that remain on master simply because they're not high intensity.

The way to figure out what is still on master is:

1. Look at the servlets only loaded on master
2. Look at replication streams on started on master
3. Look at the places in the code where we check if we're on master, usually: worker_app is None:.

[richvdh]

I'm going to close this, because the initial work is done. Remaining work includes:

• better documentation (Expand and improve workers documentation and example configuration #9072)
• getting rid of the differences between workers (Remove the concept of worker "types" #10441)
• Better testing (Synapse workers Complement image results in flaky tests due to inconsistent worker process init #10065)
• A long tail of remaining stuff which has to hit the master