Synapse fails to start with ModuleNotFoundError after upgrading to 1.60.0 when python module sqlite3 is not installed

[joh-ku]

Description

After upgrading from v1.59.1 to v1.60.0 synapse fails to start with a ModuleNotFoundError: No module named '_sqlite3' even if PostgreSQL is being used. I didn't find any hint in the release notes that this behaviour is expected.
I found a related bug report documented in #7127 and #7155.

Trace

Traceback (most recent call last):
  File "/usr/local/lib/python3.9/runpy.py", line 197, in _run_module_as_main
    return _run_code(code, main_globals, None,
  File "/usr/local/lib/python3.9/runpy.py", line 87, in _run_code
    exec(code, run_globals)
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/app/homeserver.py", line 39, in <module>
    from synapse.app import _base
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/app/_base.py", line 62, in <module>
    from synapse.events.spamcheck import load_legacy_spam_checkers
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/events/spamcheck.py", line 31, in <module>
    from synapse.rest.media.v1._base import FileInfo
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/rest/__init__.py", line 18, in <module>
    from synapse.rest import admin
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/rest/admin/__init__.py", line 29, in <module>
    from synapse.rest.admin._base import admin_patterns, assert_requester_is_admin
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/rest/admin/_base.py", line 19, in <module>
    from synapse.api.auth import Auth
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/api/auth.py", line 36, in <module>
    from synapse.storage.databases.main.registration import TokenLookupResult
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/__init__.py", line 31, in <module>
    from synapse.storage.databases import Databases
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/databases/__init__.py", line 18, in <module>
    from synapse.storage._base import SQLBaseStore
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/_base.py", line 20, in <module>
    from synapse.storage.database import make_in_list_sql_clause  # noqa: F401; noqa: F401
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/database.py", line 57, in <module>
    from synapse.storage.background_updates import BackgroundUpdater
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/background_updates.py", line 36, in <module>
    from . import engines
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/engines/__init__.py", line 18, in <module>
    from .sqlite import Sqlite3Engine
  File "/srv/synapse/system/lib/python3.9/site-packages/synapse/storage/engines/sqlite.py", line 15, in <module>
    import sqlite3
  File "/usr/local/lib/python3.9/sqlite3/__init__.py", line 57, in <module>
    from sqlite3.dbapi2 import *
  File "/usr/local/lib/python3.9/sqlite3/dbapi2.py", line 27, in <module>
    from _sqlite3 import *
ModuleNotFoundError: No module named '_sqlite3'

Steps to reproduce

• stop synapse
• upgrade synapse pip install -U matrix-synapse
• start synapse

Fix

Manual installation of databases/py-sqlite3, which is not part of the base port lang/python39, solves the issue.

(Rolling back with pip install matrix-synapse==1.59.1 also solves the issue.)

Version information

Version:

Homeserver running on v1.59.1 trying an upgrade to v1.60.0

Platform:

Python 3.9.13
PostgreSQL 13.7
FreeBSD 13.1-RELEASE

[H-Shay]

Looks like this is the result of changes in https://github.com/matrix-org/synapse/pull/12734/files#. @DMRobertson any thoughts on the most graceful way to handle this?

[DMRobertson]

I didn't know that some distributions of Python don't have sqlite3 available. It seems that we are doomed to repeat ourselves, c.f. #7127.

The easiest fix is probably to move the imports

synapse/synapse/storage/engines/__init__.py

Lines 17 to 18 in 1fe202a

	from .postgres import PostgresEngine
	from .sqlite import Sqlite3Engine

into their respective branches:

synapse/synapse/storage/engines/__init__.py

Lines 24 to 28 in 1fe202a

	if name == "sqlite3":
	return Sqlite3Engine(database_config)
	if name == "psycopg2":
	return PostgresEngine(database_config)

As a bonus, that would hopefully allow us to import psycopg2 in postgres.py without any kind of import guards.

[DMRobertson]

Sorry @H-Shay, I think I gave you bad advice here. I forgot that we import the DatabaseEngines all over the place and use them in isinstance checks.

Three options spring to mind:

1. Treat sqlite like we do psycopg2: only import it when type checking and within the methods of the SqliteEngine.
2. Something like

   try:
       from .sqlite import SqliteEngine
   except ImportError:
       class SqliteEngine(BaseDatabaseEngine):
           def __init__(self, *args, **kwargs):
               raise RuntimeError("Cannot create database engine: the sqlite3 module is not available".)

   and similarly for Postgres. See also https://github.com/matrix-org/synapse/pull/12734/files#r873691359.
3. Revert Tidy up and type-hint the database engine modules #12734.

My preferences would be 2 > 1 > 3.

[DMRobertson]

@joh-ku thanks again for spotting and reporting this regression. I believe #12979 should fix this, and that change is scheduled for Synapse 1.62. If you get the chance to confirm that'd be much appreciated!

[joh-ku]

You're welcome, thank you for the quick fix. I'll check once 1.62 is available and get back to you with the results in this thread.

[joh-ku]

@DMRobertson: Fixed with v1.62.0. Synapse starts again without having databases/py-sqlite3 installed.

[DMRobertson]

Many thanks for confirming @joh-ku!