Snapshot schema 72

[DMRobertson]

Now that

synapse/synapse/storage/schema/__init__.py

Line 15 in 44be423

SCHEMA_VERSION = 73 # remember to update the list below when updating

we can snapshot the database featuring all migrations in version <= 72. See commit-by-commit for how this is done. Had to make a few tweaks to

• the way that we ingest SQL. Our home grown SQL parser doesn't seem to understand postgres' dollar quotes.
• the schema dump script. There were some oddities in the sqlite output which only occur under .schema --indent. Easiest to drop the pretty printing :( We were also dropping some SELECT statements with side effects that we shouldn't have been.

Closes #13715.

[DMRobertson]

This seems to be a lie, because the cursor is actually a LoggingTransaction that wraps a sqlite3.Cursor.

I restrained myself from making further type annotations here.

[erikjohnston]

Out of interest have you tried comparing a pg_dump comparing before and after this?

[DMRobertson]

Out of interest have you tried comparing a pg_dump comparing before and after this?

Erm, no, but I can do.

[erikjohnston]

If its easy to check if the pg_dump before/after matches then I think its worth doing for paranoia's sake, otherwise LGTM!

[DMRobertson]

If its easy to check if the pg_dump before/after matches then I think its worth doing for paranoia's sake, otherwise LGTM!

The diff isn't very interesting:

$ diff -u synapse_{migrated,snapshot}.sql
--- synapse_migrated.sql	2022-09-26 17:44:51.297133350 +0100
+++ synapse_snapshot.sql	2022-09-26 17:39:25.482251022 +0100
@@ -2351,163 +2351,6 @@
 --
 
 COPY public.applied_schema_deltas (version, file) FROM stdin;
-55	55/access_token_expiry.sql
-55	55/track_threepid_validations.sql
-55	55/users_alter_deactivated.sql
-56	56/add_spans_to_device_lists.sql
-56	56/current_state_events_membership.sql
-56	56/current_state_events_membership_mk2.sql
-56	56/delete_keys_from_deleted_backups.sql
-56	56/destinations_failure_ts.sql
-56	56/destinations_retry_interval_type.sql.postgres
-56	56/device_stream_id_insert.sql
-56	56/devices_last_seen.sql
-56	56/drop_unused_event_tables.sql
-56	56/event_expiry.sql
-56	56/event_labels.sql
-56	56/event_labels_background_update.sql
-56	56/fix_room_keys_index.sql
-56	56/hidden_devices.sql
-56	56/nuke_empty_communities_from_db.sql
-56	56/public_room_list_idx.sql
-56	56/redaction_censor.sql
-56	56/redaction_censor2.sql
-56	56/redaction_censor3_fix_update.sql.postgres
-56	56/redaction_censor4.sql
-56	56/remove_tombstoned_rooms_from_directory.sql
-56	56/room_key_etag.sql
-56	56/room_membership_idx.sql
-56	56/room_retention.sql
-56	56/signing_keys.sql
-56	56/signing_keys_nonunique_signatures.sql
-56	56/state_group_room_idx.sql
-56	56/stats_separated.sql
-56	56/unique_user_filter_index.py
-56	56/user_external_ids.sql
-56	56/users_in_public_rooms_idx.sql
-57	57/delete_old_current_state_events.sql
-57	57/device_list_remote_cache_stale.sql
-57	57/local_current_membership.py
-57	57/remove_sent_outbound_pokes.sql
-57	57/rooms_version_column.sql
-57	57/rooms_version_column_2.sql.postgres
-57	57/rooms_version_column_3.sql.postgres
-58	58/00background_update_ordering.sql
-58	58/02remove_dup_outbound_pokes.sql
-58	58/03persist_ui_auth.sql
-58	58/05cache_instance.sql.postgres
-58	58/06dlols_unique_idx.py
-58	58/07add_method_to_thumbnail_constraint.sql.postgres
-58	58/07persist_ui_auth_ips.sql
-58	58/08_media_safe_from_quarantine.sql.postgres
-58	58/09shadow_ban.sql
-58	58/10_pushrules_enabled_delete_obsolete.sql
-58	58/10drop_local_rejections_stream.sql
-58	58/10federation_pos_instance_name.sql
-58	58/11dehydration.sql
-58	58/11fallback.sql
-58	58/11user_id_seq.py
-58	58/12room_stats.sql
-58	58/13remove_presence_allow_inbound.sql
-58	58/14events_instance_name.sql
-58	58/14events_instance_name.sql.postgres
-58	58/15_catchup_destination_rooms.sql
-58	58/15unread_count.sql
-58	58/16populate_stats_process_rooms_fix.sql
-58	58/17_catchup_last_successful.sql
-58	58/18stream_positions.sql
-58	58/19instance_map.sql.postgres
-58	58/19txn_id.sql
-58	58/20instance_name_event_tables.sql
-58	58/20user_daily_visits.sql
-58	58/21as_device_stream.sql
-58	58/21drop_device_max_stream_id.sql
-58	58/22puppet_token.sql
-58	58/22users_have_local_media.sql
-58	58/23e2e_cross_signing_keys_idx.sql
-58	58/24drop_event_json_index.sql
-58	58/25user_external_ids_user_id_idx.sql
-58	58/26access_token_last_validated.sql
-58	58/27local_invites.sql
-58	58/28drop_last_used_column.sql.postgres
-59	59/01ignored_user.py
-59	59/02shard_send_to_device.sql
-59	59/03shard_send_to_device_sequence.sql.postgres
-59	59/04_event_auth_chains.sql
-59	59/04_event_auth_chains.sql.postgres
-59	59/04drop_account_data.sql
-59	59/05cache_invalidation.sql
-59	59/06chain_cover_index.sql
-59	59/06shard_account_data.sql
-59	59/06shard_account_data.sql.postgres
-59	59/07shard_account_data_fix.sql
-59	59/08delete_pushers_for_deactivated_accounts.sql
-59	59/08delete_stale_pushers.sql
-59	59/09rejected_events_metadata.sql
-59	59/10delete_purged_chain_cover.sql
-59	59/11add_knock_members_to_stats.sql
-59	59/11drop_thumbnail_constraint.sql.postgres
-59	59/12account_validity_token_used_ts_ms.sql
-59	59/12presence_stream_instance.sql
-59	59/12presence_stream_instance_seq.sql.postgres
-59	59/13users_to_send_full_presence_to.sql
-59	59/14refresh_tokens.sql
-59	59/15locks.sql
-59	59/16federation_inbound_staging.sql
-60	60/01recreate_stream_ordering.sql.postgres
-60	60/02change_stream_ordering_columns.sql.postgres
-61	61/01change_appservices_txns.sql.postgres
-61	61/01insertion_event_lookups.sql
-61	61/02drop_redundant_room_depth_index.sql
-61	61/02state_groups_state_n_distinct.sql.postgres
-61	61/03recreate_min_depth.py
-62	62/01insertion_event_extremities.sql
-63	63/01create_registration_tokens.sql
-63	63/02delete_unlinked_email_pushers.sql
-63	63/02populate-rooms-creator.sql
-63	63/03session_store.sql
-63	63/04add_presence_stream_not_offline_index.sql
-64	64/01msc2716_chunk_to_batch_rename.sql.postgres
-65	65/01msc2716_insertion_event_edges.sql
-65	65/03remove_hidden_devices_from_device_inbox.sql
-65	65/04_local_group_updates.sql
-65	65/05_remove_room_stats_historical_and_user_stats_historical.sql
-65	65/06remove_deleted_devices_from_device_inbox.sql
-65	65/07_arbitrary_relations.sql
-65	65/08_device_inbox_background_updates.sql
-65	65/10_expirable_refresh_tokens.sql
-65	65/11_devices_auth_provider_session.sql
-67	67/01drop_public_room_list_stream.sql
-68	68/01event_columns.sql
-68	68/02_msc2409_add_device_id_appservice_stream_type.sql
-68	68/03_delete_account_data_for_deactivated_accounts.sql
-68	68/04_refresh_tokens_index_next_token_id.sql
-68	68/04partial_state_rooms.sql
-68	68/05partial_state_rooms_triggers.py
-68	68/06_msc3202_add_device_list_appservice_stream_type.sql
-69	69/01as_txn_seq.py
-69	69/01device_list_oubound_by_room.sql
-69	69/02cache_invalidation_index.sql
-70	70/01clean_table_purged_rooms.sql
-70	70/08_state_group_edges_unique.sql
-71	71/01rebuild_event_edges.sql.postgres
-71	71/01remove_noop_background_updates.sql
-71	71/02event_push_summary_unique.sql
-72	72/01add_room_type_to_state_stats.sql
-72	72/01event_push_summary_receipt.sql
-72	72/02event_push_actions_index.sql
-72	72/03bg_populate_events_columns.py
-72	72/03drop_event_reference_hashes.sql
-72	72/03remove_groups.sql
-72	72/04drop_column_application_services_state_last_txn.sql.postgres
-72	72/05receipts_event_stream_ordering.sql
-72	72/05remove_unstable_private_read_receipts.sql
-72	72/06add_consent_ts_to_users.sql
-72	72/06thread_notifications.sql
-72	72/07force_update_current_state_events_membership.py
-72	72/07thread_receipts.sql.postgres
-72	72/08begin_cache_invalidation_seq_at_2.sql.postgres
-72	72/08thread_receipts.sql
 \.

So I think this looks good?

Script

docker run --rm -e POSTGRES_PASSWORD=postgres -e POSTGRES_USER=postgres -e POSTGRES_DB=synapse -p 5432:5432 postgres:10-alpine

rm -r synapse/storage/schema/main/delta/73

cat >/tmp/db.yaml <<EOF
server_name: "test"
signing_key_path: "/tmp/key"

macaroon_secret_key: "abcde"

report_stats: false

databases:
  postgres:
    name: "psycopg2"
    data_stores: ["main", "state"]
    args:
      user: "postgres"
      host: "127.0.0.1"
      password: postgres
      database: synapse_snapshot

# Suppress the key server warning.
trusted_key_servers: []
EOF


python -m synapse.app.homeserver --generate-keys -c /tmp/db.yaml
PGPASSWORD=postgres createdb -U postgres -h 127.0.0.1 --lc-collate=C --lc-ctype=C --template=template0 synapse_snapshot
update_synapse_database --run-background-updates --database-config db.yaml -v
PGPASSWORD=postgres pg_dump -h 127.0.0.1 -U postgres synapse_snapshot > synapse_snapshot.sql

rm -r synapse/storage/schema/{common,main,state}/full_schemas/72/
sed -i /tmp/db.yaml -e "s/database: synapse_snapshot/database: synapse_migrated/"
PGPASSWORD=postgres createdb -U postgres -h 127.0.0.1 --lc-collate=C --lc-ctype=C --template=template0 synapse_migrated
update_synapse_database --run-background-updates --database-config db.yaml -v
PGPASSWORD=postgres pg_dump -h 127.0.0.1 -U postgres synapse_migrated > synapse_migrated.sql

diff -u synapse_{migrated,snapshot}

[erikjohnston]

Yup, I think that looks right