Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

tell mysql to ignore the sort index for search queries #32074

Merged
merged 1 commit into from
Apr 25, 2022

Conversation

icewind1991
Copy link
Member

Cleaner version of #29300 (ab)using mysql's inability to use an index for sorting by function.

mysql really likes to pick an index for sorting if it can't fully satisfy the where
filter with an index, since search queries pretty much never are fully filtered by index
mysql often picks an index for sorting instead of the much more useful index for filtering.

Signed-off-by: Robin Appelman <robin@icewind.nl>
Copy link
Member

@CarlSchwan CarlSchwan left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

ugly but makes sense :(

@icewind1991 icewind1991 merged commit f7413b9 into master Apr 25, 2022
@icewind1991 icewind1991 deleted the mysql-search-ignore-index-2 branch April 25, 2022 11:23
@icewind1991
Copy link
Member Author

/backport to stable24

@icewind1991
Copy link
Member Author

/backport to stable23

@icewind1991
Copy link
Member Author

/backport to stable22

@bdrewery
Copy link

bdrewery commented Sep 16, 2022

This kills performance for file searches (likes /apps/photos [WITH PREVIEWS EXISTING]) on MariaDB 10.5.

root@localhost [nextcloud]> EXPLAIN SELECT `file`.`fileid`, `storage`, `path`, `path_hash`, `file`.`parent`, `name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `permissions`, `checksum`, `metadata_etag`, `creation_time`, `upload_time`, `unencrypted_size` FROM `oc_filecache` `file` LEFT JOIN `oc_filecache_extended` `fe` ON `file`.`fileid` = `fe`.`fileid` WHERE ((`mimetype` = 17) OR (`mimetype` = 7) OR (`mimetype` = 22) OR (`mimetype` = 12) OR (`mimetype` = 23)) AND (((`storage` = 3) AND ((`path` = 'files') OR (`path` LIKE 'files/%'))) OR ((`storage` = 1) AND ((`path_hash` = 'e46cc72327dfc3ccaf32f0a167e6c6d1') OR (`path` LIKE '__groupfolders/1/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared') OR (`path` LIKE 'files/Shared/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared Family') OR (`path` LIKE 'files/Shared Family/%')))) ORDER BY `mtime` + '0' desc LIMIT 40;
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+------------------------+---------+-----------------------+--------+----------------------------------------------------+
| id   | select_type | table | type   | possible_keys                                                                                                        | key                    | key_len | ref                   | rows   | Extra                                              |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+------------------------+---------+-----------------------+--------+----------------------------------------------------+
|    1 | SIMPLE      | file  | range  | fs_storage_path_hash,fs_storage_mimetype,fs_storage_mimepart,fs_storage_size,fs_storage_path_prefix,fs_path_mimetype | fs_storage_path_prefix | 267     | NULL                  | 522013 | Using index condition; Using where; Using filesort |
|    1 | SIMPLE      | fe    | eq_ref | PRIMARY                                                                                                              | PRIMARY                | 8       | nextcloud.file.fileid | 1      | Using where                                        |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+------------------------+---------+-----------------------+--------+----------------------------------------------------+
2 rows in set (0.005 sec)
[run query]
40 rows in set (8.793 sec)

Without the + '0' added on it doesn't scan through 500k rows in a temp table.

root@localhost [nextcloud]> EXPLAIN SELECT `file`.`fileid`, `storage`, `path`, `path_hash`, `file`.`parent`, `name`, `mimetype`, `mimepart`, `size`, `mtime`, `storage_mtime`, `encrypted`, `etag`, `permissions`, `checksum`, `metadata_etag`, `creation_time`, `upload_time`, `unencrypted_size` FROM `oc_filecache` `file` LEFT JOIN `oc_filecache_extended` `fe` ON `file`.`fileid` = `fe`.`fileid` WHERE ((`mimetype` = 17) OR (`mimetype` = 7) OR (`mimetype` = 22) OR (`mimetype` = 12) OR (`mimetype` = 23)) AND (((
`storage` = 3) AND ((`path` = 'files') OR (`path` LIKE 'files/%'))) OR ((`storage` = 1) AND ((`path_hash` = 'e46cc72327dfc3ccaf32f0a167e6c6d1') OR (`path` LIKE '__groupfolders/1/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared') OR (`path` LIKE 'files/Shared/%'))) OR ((`storage` = 7) AND ((`path` = 'files/Shared Family') OR (`path` LIKE 'files/Shared Family/%')))) ORDER BY `mtime` desc LIMIT 40;
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+----------+---------+-----------------------+------+-------------+
| id   | select_type | table | type   | possible_keys                                                                                                        | key      | key_len | ref                   | rows | Extra       |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+----------+---------+-----------------------+------+-------------+
|    1 | SIMPLE      | file  | index  | fs_storage_path_hash,fs_storage_mimetype,fs_storage_mimepart,fs_storage_size,fs_storage_path_prefix,fs_path_mimetype | fs_mtime | 8       | NULL                  | 69   | Using where |
|    1 | SIMPLE      | fe    | eq_ref | PRIMARY                                                                                                              | PRIMARY  | 8       | nextcloud.file.fileid | 1    | Using where |
+------+-------------+-------+--------+----------------------------------------------------------------------------------------------------------------------+----------+---------+-----------------------+------+-------------+
2 rows in set (0.005 sec)
[flush cache and run query without + '0']
40 rows in set (1.168 sec)

Reverting the change allows my /apps/photos to be useable rather than 10 second hangs when scrolling.

@icewind1991
Copy link
Member Author

From what I've seen in large instances, the fs_storage_path_prefix is usually a lot more significant in speedups than the fs_mtime index. Using the mtime index is only really useful if most of the (recent) files belong to the user performing the search, and if the there are significantly more search results than the LIMIT, else it just ends up scanning most of the table sorted by mtime anyway.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants