federation sender doesn't work when invoked as synapse.app.generic_worker

[maranda]

Description

As in the title, I followed the documentation closely to setup the worker, but when a message is propagated through Federation servers will answer with 403s

If I leave federation sending to the master instance there're no issues.

Steps to reproduce

• Send message in a room
• Message appears locally but is not replicated over the network
• Remote servers answer 403s to federation transactions

Messages appear locally but not remotely and in logs the following warnings appear:
2020-08-01 11:29:46,696 - synapse.http.matrixfederationclient - 536 - WARNING - federation_transaction_transmission_loop-4108 - {PUT-O-3160} [***] Request failed: PUT matrix://***/_matrix/federation/v1/send/1596280923582: HttpResponseException("403: b'Forbidden'")

Version information

• Homeserver: aria-net.org

If not matrix.org:

• Version: 1.18

• Install method: Ubuntu packages

• Platform:
  Hyper-V machine with 16GB of RAM and eight virtual processors
  Synapse running with 1+3 workers (2 general, 1 federation sender) with Postgresql and Redis managed via systemd
  OS: Ubuntu server 20.04

[maranda]

I found the issue.

Basically https://github.com/matrix-org/synapse/blob/develop/docs/systemd-with-workers/system/matrix-synapse-worker%40.service will only startup the generic worker I didn't really look inside the script so I didn't notice it.

I think that should be properly referenced in the documentation, that the script should be modified to use other worker typed e.g. synapse.app.federation_sender.

[erikjohnston]

Thanks for the bug report! Part of me feels like we just make everything use the same app to avoid this problem, but until then yeah let's just fix the documentation 👍

[richvdh]

I'd so much rather fix the app than the docs...

[richvdh]

I'm confused by this. Why does the service file need changing? Why doesn't it work when it starts -m synapse.app.generic_worker ?

[maranda]

@richvdh it's not that the worker didn't start it's that the generic worker didn't work at all for the federation sender role. And I had to start the synapse.app.federation_sender directly via the systemd service configuration files to rectify that.

[richvdh]

yes; I'm asking why that is the case. AFAICT the code that gets run is identical.

[richvdh]

(that's mostly a question for the dev team!)

[clokep]

There's still some special-cased code that needs to get cleaned-up:

synapse/synapse/app/generic_worker.py

Lines 952 to 966 in 19b15d6

	if config.worker_app == "synapse.app.federation_sender":
	if config.worker.send_federation:
	sys.stderr.write(
	"\nThe send_federation must be disabled in the main synapse process"
	"\nbefore they can be run in a separate worker."
	"\nPlease add ``send_federation: false`` to the main config"
	"\n"
	)
	sys.exit(1)
	# Force the pushers to start since they will be disabled in the main config
	config.worker.send_federation = True
	else:
	# For other worker types we force this to off.
	config.worker.send_federation = False

[erikjohnston]

Hmm, actually starting it with -m synapse.app.generic_worker should worker? It needs worker_app: synapse.app.federation_sender set in the config though: https://github.com/matrix-org/synapse/blob/develop/synapse/app/generic_worker.py#L952-L963 . Maybe we're doing something cunning somewhere that ensure we're setting worker_app to the same as the started app?

[maranda]

@erikjohnston worker_app: synapse.app.federation_sender was and is set in the config but something breaks when the generic worker loads the federation sender app, that's the problem, I had to revolve on modifying the service files to load the app directly to counter that.

[richvdh]

basically, I don't think this is a documentation bug. If it exists at all, it's an implementation bug.

[clokep]

We think that #9466 will fix this, which will be part of Synapse v1.29.0.