/sync fails with a RuntimeError: Database corruption due to an apparently modified event

[Prince31]

Description

My synapse matrix was working fine with version 1.32.0rc1. Now I upgraded to version 1.65 after which few of my users are getting Internal Server Error in sync. Getting error:

Steps to reproduce

• list the steps
• that reproduce the bug
• using hyphens as bullet points
• upgrade synapse version from v.132.0rc1 to 1.65
• observe Internal Server Error in sync

Homeserver

another homeserver

Synapse Version

1,65

Installation Method

pip (from PyPI)

Database

PostgreSQL 12

Workers

Multiple workers

Platform

Ubuntu 16.04
VM

Configuration

Presence: true

Relevant log output

res = await self.response_cache.wrap(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/util/caches/response_cache.py",
line 265, in wrap
     return await make_deferred_yieldable(result)
   File
"/root/matrix-backend-repo/env_upgrade/lib/python3.8/site-packages/twisted/internet/defer.py",
line 1693, in _inlineCallbacks
     result = context.run(
   File
"/root/matrix-backend-repo/env_upgrade/lib/python3.8/site-packages/twisted/python/failure.py",
line 518, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/util/caches/response_cache.py",
line 246, in cb
     return await callback(*args, **kwargs)
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 344, in _wait_for_sync_for_user
     result: SyncResult = await self.current_sync_for_user(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 395, in current_sync_for_user
     sync_result = await self.generate_sync_result(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 1124, in generate_sync_result
     res = await self._generate_sync_entry_for_rooms(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 1573, in _generate_sync_entry_for_rooms
     await concurrently_execute(handle_room_entries, room_entries, 10)
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/util/async_helpers.py",
line 235, in concurrently_execute
     await yieldable_gather_results(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/util/async_helpers.py",
line 291, in yieldable_gather_results
     raise dfe.subFailure.value from None
   File
"/root/matrix-backend-repo/env_upgrade/lib/python3.8/site-packages/twisted/internet/defer.py",
line 1693, in _inlineCallbacks
     result = context.run(
   File
"/root/matrix-backend-repo/env_upgrade/lib/python3.8/site-packages/twisted/python/failure.py",
line 518, in throwExceptionIntoGenerator
     return g.throw(self.type, self.value, self.tb)
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/util/async_helpers.py",
line 227, in _concurrently_execute_inner
     await maybe_awaitable(func(value))
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 1562, in handle_room_entries
     await self._generate_room_entry(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 2010, in _generate_room_entry
     batch = await self._load_filtered_recents(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/handlers/sync.py",
line 565, in _load_filtered_recents
     events, end_key = await self.store.get_recent_events_for_room(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/storage/databases/main/stream.py",
line 696, in get_recent_events_for_room
     events = await self.get_events_as_list(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/storage/databases/main/events_worker.py",
line 471, in get_events_as_list
     event_entry_map = await self._get_events_from_cache_or_db(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/storage/databases/main/events_worker.py",
line 732, in _get_events_from_cache_or_db
     missing_events: Dict[str, EventCacheEntry] = await delay_cancellation(
   File
"/root/matrix-backend-repo/env_upgrade/lib/python3.8/site-packages/twisted/internet/defer.py",
line 1697, in _inlineCallbacks
     result = context.run(gen.send, result)
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/storage/databases/main/events_worker.py",
line 717, in get_missing_events_from_cache_or_db
     raise e
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/storage/databases/main/events_worker.py",
line 709, in get_missing_events_from_cache_or_db
     db_missing_events = await self._get_events_from_db(
   File
"/root/matrix-backend-repo/riot-matrix/synapse/synapse/storage/databases/main/events_worker.py",
line 1256, in _get_events_from_db
     raise RuntimeError(
RuntimeError: Database corruption: Event
$nXceavnppbYIu7U3E0T57KUA_ZCG1XPkw_tCBR8H6pY in room
!RSLcBHqojDbZdVVwpV:matrix.hakase-labs.io from the database appears to
have been modified (calculated event id
$bGeuLBLWC-yfxSMKYmcQwO0-SuhTMcAF5PjZBup7HS8)

Anything else that would be useful to know?

No response

[H-Shay]

Detection for database corruption was added in Synapse 1.59.0rc1 (2022-05-10), namely in #12620. It's hard to know if the event was recently corrupted, or if your upgrading to the Synapse version with the check just uncovered it. Of interest to you might be the discussion at #12837.

Do you or have you ever had retention enabled on this homeserver?

[H-Shay]

This first step to resolving this would be to try purging the room the affected event came from and see if that resolves it.

[H-Shay]

I am going to close this issue for now due to lack of activity.