Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add another index to device_lists_changes_in_room #14534

Merged
merged 2 commits into from
Nov 23, 2022
Merged

Add another index to device_lists_changes_in_room #14534

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
3754bc5
Add another index to `device_lists_changes_in_room`
erikjohnston Nov 23, 2022
da918f5
Newsfile
erikjohnston Nov 23, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions changelog.d/14534.misc
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
Improve DB performance by reducing amount of data that gets read in `device_lists_changes_in_room`.
7 changes: 7 additions & 0 deletions synapse/storage/databases/main/devices.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -1441,6 +1441,13 @@ def __init__(
self._remove_duplicate_outbound_pokes,
)

self.db_pool.updates.register_background_index_update(
"device_lists_changes_in_room_by_room_index",
index_name="device_lists_changes_in_room_by_room_idx",
table="device_lists_changes_in_room",
columns=["room_id", "stream_id"],
Comment on lines +1447 to +1448
Copy link
Contributor

@MadLittleMods MadLittleMods Nov 24, 2022 •
edited
Loading

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Was #14516 some prerequisite work for this change (or at least to see a performance difference with the combination)?

From a glance, it seems #14516 changed the lookup to use (stream_id, room_id) and this PR adds an index for that

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it was a combination really, but hard to know. The main aim of #14516 was to make it so that the rows of device_lists_changes_in_room on disk were roughly ordered by stream ID, making it much more efficient to look at rows in stream ID order.

This PR mainly speeds up queries that were being done during sync, around "has anything changed since X in a particular set of rooms R"

Comment on lines +1447 to +1448
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It looks like this made a big difference to database disk reads when it was deployed to matrix.org😮

https://grafana.matrix.org/d/000000009/postgres?orgId=1&refresh=1m&var-data_source=Prometheus&var-database=matrix&var-bucket_size=%24__auto_interval_bucket_size&var-instance=manticore.matrix.org%3A9113&from=1668993778198&to=1669252978198&viewPanel=51

)

async def _drop_device_list_streams_non_unique_indexes(
self, progress: JsonDict, batch_size: int
) -> int:
Expand Down
20 changes: 20 additions & 0 deletions synapse/storage/schema/main/delta/73/13add_device_lists_index.sql
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,20 @@
/* Copyright 2022 The Matrix.org Foundation C.I.C
*
* Licensed under the Apache License, Version 2.0 (the "License");
* you may not use this file except in compliance with the License.
* You may obtain a copy of the License at
*
* http://www.apache.org/licenses/LICENSE-2.0
*
* Unless required by applicable law or agreed to in writing, software
* distributed under the License is distributed on an "AS IS" BASIS,
* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
* See the License for the specific language governing permissions and
* limitations under the License.
*/


-- Adds an index on `device_lists_changes_in_room (room_id, stream_id)`, which
-- speeds up `/sync` queries.
INSERT INTO background_updates (ordering, update_name, progress_json) VALUES
(7313, 'device_lists_changes_in_room_by_room_index', '{}');