Allow producer-consumer pattern for workers

[ShadowJonathan]

Currently, most workers (e.g. federation sender) are locked to a specific amount and specifically-named workers, this limits the options of deployment for synapse quite a bit, and actually makes its deployment more fragile, as if a worker fails, and the synapse homeserver instance is expecting it to exist, as the federation sender documentation describes; it'll drop events.

To change this to a dynamic producer-consumer pattern will allow for things such as stateless dynamic scaling, where a supervisor will judge where or when another worker instance will be spawned to handle more load, this can scale synapse/matrix significantly without much cost.

Maybe it's also needed to switch from redis to rabbitmq, which allows for a much clearer and integrated method of consumer-producer (with acknowledgements allowing for robust fallback and retries of failed jobs), aio-pika can help with this, as a modern asynchronous library that works with rabbitmq.

[clokep]

I think this is mixing up a problem statement and potential solution. It sounds like you want to enable dynamic scaling and are running into issues?

For the most part each worker should be able to be of the same time now (the "generic worker"), but there is some configuration to tell workers how to find itself...this could be replaced with a more general service discovery mechanism, but that was punted until it was needed.

The worker documentation doesn't make it abundantly clear that the names given there don't really need to be part of the config anymore (at least that's my understanding). 😢

[ShadowJonathan]

It sounds like you want to enable dynamic scaling and are running into issues?

No, i'm describing that i have had some experience with workers now, and i see it can be improved. (as the current solution doesn't allow it)

The worker documentation doesn't make it abundantly clear that the names given there don't really need to be part of the config anymore (at least that's my understanding). cry

...then that can be part of #9072, with which i already said that the worker documentation could receive some love and improvement in general.

I suggested rabbitmq because I think it'd be interesting to look at for a solution for this, my problem statement is more "i cannot dynamically add or remove workers (as a principle)", and so working to move to rabbitmq could open up synapse for better scalability alongside it (maybe even allow to separate core components into multiple queue consumers, and/or give each sync stream a seperate queue, just to name some potential improvements)

[richvdh]

I don't think this is very actionable. Some worker functions are already dynamically scalable; those that aren't have more fundamental problems than whether or not we use rabbitmq.