Unable to leave room. KeyError

[TheNamelessWonderer]

Description

When I try to leave a room it fails with internal server error 500. I am also not get any new messages in the channel.

https://matrix.server.org/_matrix/client/r0/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/leave

MatrixError: [500] Internal server error

Steps to reproduce

• Try to leave the room
• See error

Homeserver

My own HomeServer

Synapse Version

1.86.0rc2

Installation Method

Docker (matrixdotorg/synapse)

Database

PostgresSQL

Workers

Single process

Platform

Debian 11 in docker.

i5 and 32GB of RAM, SSD.

Configuration

Presence is enabled.

Relevant log output

2023-06-18 18:40:10,537 - synapse.http.server - 133 - ERROR - POST-8118 - Failed handle request via 'RoomMembershipRestServlet': <XForwardedForRequest at 0x7f21f5d05fd0 method='POST' uri='/_matrix/client/r0/rooms/!ehXvUhWNASUkSLvAGP%3Amatrix.org/leave' clientproto='HTTP/1.0' site='8008'>
Traceback (most recent call last):
  File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 319, in _async_render_wrapper
    callback_return = await self._async_render(request)
                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/http/server.py", line 529, in _async_render
    callback_return = await raw_callback_return
                      ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1072, in on_POST
    return await self._do(request, requester, room_id, membership_action, None)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/rest/client/room.py", line 1045, in _do
    await self.room_member_handler.update_membership(
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 657, in update_membership
    result = await self.update_membership_locked(
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 1127, in update_membership_locked
    return await self._local_membership_update(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/room_member.py", line 508, in _local_membership_update
    await self.event_creation_handler.handle_new_client_event(
  File "/usr/local/lib/python3.11/site-packages/synapse/util/metrics.py", line 113, in measured_func
    r = await func(self, *args, **kwargs)
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1492, in handle_new_client_event
    result, _ = await make_deferred_yieldable(
                ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1697, in _inlineCallbacks
    result = context.run(gen.send, result)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1571, in _persist_events
    event = await self.persist_and_notify_client_events(
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/handlers/message.py", line 1988, in persist_and_notify_client_events
    ) = await self._storage_controllers.persistence.persist_events(
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/logging/opentracing.py", line 916, in _wrapper
    return await func(*args, **kwargs)  # type: ignore[misc]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 413, in persist_events
    ret_vals = await yieldable_gather_results(enqueue, partitioned.items())
               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/util/async_helpers.py", line 304, in yieldable_gather_results
    raise dfe.subFailure.value from None
  File "/usr/local/lib/python3.11/site-packages/twisted/internet/defer.py", line 1693, in _inlineCallbacks
    result = context.run(
             ^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/twisted/python/failure.py", line 518, in throwExceptionIntoGenerator
    return g.throw(self.type, self.value, self.tb)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 408, in enqueue
    return await self._event_persist_queue.add_to_queue(
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 239, in add_to_queue
    res = await make_deferred_yieldable(end_item.deferred.observe())
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 282, in handle_queue_loop
    ret = await self._per_item_callback(room_id, item.task)
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 354, in _process_event_persist_queue_task
    return await self._persist_event_batch(room_id, task)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 672, in _persist_event_batch
    res = await self._get_new_state_after_events(
          ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 875, in _get_new_state_after_events
    old_state_groups = {
                       ^
  File "/usr/local/lib/python3.11/site-packages/synapse/storage/controllers/persist_events.py", line 876, in <setcomp>
    event_id_to_state_group[evid] for evid in old_latest_event_ids
    ~~~~~~~~~~~~~~~~~~~~~~~^^^^^^
KeyError: '$4VUpqvdeRHc3Ci-3Bjn9yh0o4KaGFK3MPQXQrAFtEkc'

Anything else that would be useful to know?

I haven't add any new messages in the channel since April and this was an issue on 1.85.2 as well.

[erikjohnston]

@TheNamelessWonderer have you deleted/purged any events in the room with the admin API?

It looks like somehow we don't have state for the old forwards extremity.

[TheNamelessWonderer]

I haven’t purged events but I might have purged the whole room from the database once and rejoined it. If I try to purge it now I just get an error and it doesn’t remove the room from the database because it can’t kick the users I presume.

[MadLittleMods]

If you've purged the room before, this is probably another case of #13476 (see this comment specifically for the exact same error when trying to leave)

I don't know how to resolve your specific scenario but this is an area which we're actively working on fixes so this kind of thing won't happen in the first place.

Judging from past corruption issues like #12507, I'm going to close in favor of #13476

Also related to #12821