Synapse 1.100rc1 makes Postgres eat the whole CPU

[tulir]

Description

Didn't happen immediately after upgrade, I'm not exactly sure what triggered it. #16833 sounds relevant since postgres active queries had lots of FROM event_auth_chain_links WHERE origin_chain_id = ANY(ARRAY[...]). Postgres was using the whole CPU by having multiple threads running similar queries.

After restarting synapse it calmed down slightly, but at least one postgres thread started running those queries again and immediately used 100% of a core, so I downgraded to 1.99 to avoid the whole CPU being eaten again overnight.

Steps to reproduce

1. Upgrade to Synapse 1.100rc1
2. Wait a while?
3. Observe the system load exploding

Homeserver

maunium.net

Synapse Version

1.100rc1

Installation Method

Docker (dock.mau.dev/maunium/synapse)

Database

PostgreSQL

Workers

Multiple workers

Platform

Docker on Ubuntu 22.04

[erikjohnston]

Hopefully fixed by #16849