SerializationFailure when rotating push actions

[erikjohnston]

We recently changed things so we rotate notifications almost immediately, rather than waiting a day. This interacts badly with people sending read receipts, as that transaction attempts to modify the push summary tables.

We need to fix this or revert before we make another release.

Broke by #13005

Stack trace:

SerializationFailure: could not serialize access due to concurrent update

  File "synapse/metrics/background_process_metrics.py", line 240, in run
    return await func(*args, **kwargs)
  File "synapse/storage/databases/main/event_push_actions.py", line 806, in _rotate_notifs
    caught_up = await self.db_pool.runInteraction(
  File "synapse/storage/database.py", line 835, in runInteraction
    return await delay_cancellation(_runInteraction())
  File "twisted/internet/defer.py", line 1656, in _inlineCallbacks
    result = current_context.run(
  File "twisted/python/failure.py", line 514, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
  File "synapse/storage/database.py", line 807, in _runInteraction
    result = await self.runWithConnection(
  File "synapse/storage/database.py", line 930, in runWithConnection
    return await make_deferred_yieldable(
  File "twisted/python/threadpool.py", line 244, in inContext
    result = inContext.theWork()  # type: ignore[attr-defined]
  File "twisted/python/threadpool.py", line 260, in <lambda>
    inContext.theWork = lambda: context.call(  # type: ignore[attr-defined]
  File "twisted/python/context.py", line 117, in callWithContext
    return self.currentContext().callWithContext(ctx, func, *args, **kw)
  File "twisted/python/context.py", line 82, in callWithContext
    return func(*args, **kw)
  File "twisted/enterprise/adbapi.py", line 282, in _runWithConnection
    result = func(conn, *args, **kw)
  File "synapse/storage/database.py", line 923, in inner_func
    return func(db_conn, *args, **kwargs)
  File "synapse/storage/database.py", line 671, in new_transaction
    r = func(cursor, *args, **kwargs)
  File "synapse/storage/databases/main/event_push_actions.py", line 850, in _rotate_notifs_txn
    self._rotate_notifs_before_txn(txn, rotate_to_stream_ordering)
  File "synapse/storage/databases/main/event_push_actions.py", line 950, in _rotate_notifs_before_txn
    txn.execute_batch(
  File "synapse/storage/database.py", line 329, in execute_batch
    self._do_execute(
  File "synapse/storage/database.py", line 395, in _do_execute
    return func(sql, *args, **kwargs)
  File "synapse/storage/database.py", line 330, in <lambda>
    lambda the_sql: execute_batch(self.txn, the_sql, args), sql
  File "psycopg2/extras.py", line 1187, in execute_batch
    cur.execute(b";".join(sqls))

Sentry: https://sentry.tools.element.io/organizations/element/issues/924/